Adding functionality with extensions

bos@559: bos@559: bos@559: bos@572: bos@559: Adding functionality with extensions bos@559: bos@584: While the core of Mercurial is quite complete from a bos@559: functionality standpoint, it's deliberately shorn of fancy bos@559: features. This approach of preserving simplicity keeps the bos@559: software easy to deal with for both maintainers and users. bos@559: bos@584: However, Mercurial doesn't box you in with an inflexible bos@559: command set: you can add features to it as bos@559: extensions (sometimes known as bos@559: plugins). We've already discussed a few of bos@559: these extensions in earlier chapters. bos@559: bos@592: bos@559: covers the fetch extension; bos@559: this combines pulling new changes and merging them with local bos@559: changes into a single command, fetch. bos@559: bos@592: In , we covered bos@559: several extensions that are useful for hook-related bos@559: functionality: acl adds bos@559: access control lists; bugzilla adds integration with the bos@559: Bugzilla bug tracking system; and notify sends notification emails on bos@559: new changes. bos@559: bos@584: The Mercurial Queues patch management extension is bos@559: so invaluable that it merits two chapters and an appendix all bos@592: to itself. covers the bos@592: basics; discusses advanced topics; bos@592: and goes into detail on bos@559: each bos@559: command. bos@559: bos@559: bos@584: In this chapter, we'll cover some of the other extensions that bos@559: are available for Mercurial, and briefly touch on some of the bos@559: machinery you'll need to know about if you want to write an bos@559: extension of your own. bos@559: bos@592: In , bos@559: we'll discuss the possibility of huge bos@559: performance improvements using the inotify extension. bos@559: bos@559: bos@559: bos@559: Improve performance with the <literal bos@559: role="hg-ext">inotify</literal> extension bos@559: bos@584: Are you interested in having some of the most common bos@559: Mercurial operations run as much as a hundred times faster? bos@559: Read on! bos@559: bos@584: Mercurial has great performance under normal circumstances. bos@559: For example, when you run the hg bos@559: status command, Mercurial has to scan almost every bos@559: directory and file in your repository so that it can display bos@559: file status. Many other Mercurial commands need to do the same bos@559: work behind the scenes; for example, the hg diff command uses the status bos@559: machinery to avoid doing an expensive comparison operation on bos@559: files that obviously haven't changed. bos@559: bos@584: Because obtaining file status is crucial to good bos@559: performance, the authors of Mercurial have optimised this code bos@559: to within an inch of its life. However, there's no avoiding the bos@559: fact that when you run hg bos@559: status, Mercurial is going to have to perform at bos@559: least one expensive system call for each managed file to bos@559: determine whether it's changed since the last time Mercurial bos@559: checked. For a sufficiently large repository, this can take a bos@559: long time. bos@559: bos@584: To put a number on the magnitude of this effect, I created a bos@559: repository containing 150,000 managed files. I timed hg status as taking ten seconds to bos@559: run, even when none of those files had been bos@559: modified. bos@559: bos@584: Many modern operating systems contain a file notification bos@559: facility. If a program signs up to an appropriate service, the bos@559: operating system will notify it every time a file of interest is bos@559: created, modified, or deleted. On Linux systems, the kernel bos@559: component that does this is called bos@559: inotify. bos@559: bos@584: Mercurial's inotify bos@559: extension talks to the kernel's inotify bos@559: component to optimise hg status bos@559: commands. The extension has two components. A daemon sits in bos@559: the background and receives notifications from the bos@559: inotify subsystem. It also listens for bos@559: connections from a regular Mercurial command. The extension bos@672: modifies Mercurial's behavior so that instead of scanning the bos@559: filesystem, it queries the daemon. Since the daemon has perfect bos@559: information about the state of the repository, it can respond bos@559: with a result instantaneously, avoiding the need to scan every bos@559: directory and file in the repository. bos@559: bos@584: Recall the ten seconds that I measured plain Mercurial as bos@559: taking to run hg status on a bos@559: 150,000 file repository. With the inotify extension enabled, the time bos@559: dropped to 0.1 seconds, a factor of one bos@559: hundred faster. bos@559: bos@584: Before we continue, please pay attention to some bos@559: caveats. bos@559: bos@584: The inotify bos@559: extension is Linux-specific. Because it interfaces directly bos@559: to the Linux kernel's inotify subsystem, bos@559: it does not work on other operating systems. bos@559: bos@584: It should work on any Linux distribution that bos@559: was released after early 2005. Older distributions are bos@559: likely to have a kernel that lacks bos@559: inotify, or a version of bos@559: glibc that does not have the necessary bos@559: interfacing support. bos@559: bos@584: Not all filesystems are suitable for use with bos@559: the inotify extension. bos@559: Network filesystems such as NFS are a non-starter, for bos@559: example, particularly if you're running Mercurial on several bos@559: systems, all mounting the same network filesystem. The bos@559: kernel's inotify system has no way of bos@559: knowing about changes made on another system. Most local bos@559: filesystems (e.g. ext3, XFS, ReiserFS) should work bos@559: fine. bos@559: bos@559: bos@584: The inotify extension is bos@559: not yet shipped with Mercurial as of May 2007, so it's a little bos@559: more involved to set up than other extensions. But the bos@559: performance improvement is worth it! bos@559: bos@584: The extension currently comes in two parts: a set of patches bos@559: to the Mercurial source code, and a library of Python bindings bos@559: to the inotify subsystem. bos@559: bos@584: There are two Python bos@559: inotify binding libraries. One of them is bos@559: called pyinotify, and is packaged by some bos@559: Linux distributions as python-inotify. bos@559: This is not the one you'll need, as it is bos@559: too buggy and inefficient to be practical. bos@559: bos@584: To get going, it's best to already have a functioning copy bos@559: of Mercurial installed. bos@559: bos@584: If you follow the instructions below, you'll be bos@559: replacing and overwriting any existing bos@559: installation of Mercurial that you might already have, using bos@559: the latest bleeding edge Mercurial code. Don't bos@559: say you weren't warned! bos@559: bos@559: bos@584: Clone the Python inotify bos@559: binding repository. Build and install it. bos@580: hg clone http://hg.kublai.com/python/inotify bos@580: cd inotify bos@580: python setup.py build --force bos@580: sudo python setup.py install --skip-build bos@559: bos@584: Clone the crew Mercurial repository. bos@559: Clone the inotify patch bos@559: repository so that Mercurial Queues will be able to apply bos@559: patches to your cope of the crew repository. bos@580: hg clone http://hg.intevation.org/mercurial/crew bos@580: hg clone crew inotify bos@580: hg clone http://hg.kublai.com/mercurial/patches/inotify inotify/.hg/patches bos@559: bos@584: Make sure that you have the Mercurial Queues bos@559: extension, mq, enabled. If bos@592: you've never used MQ, read to get started bos@559: quickly. bos@559: bos@584: Go into the inotify repo, and apply all bos@559: of the inotify patches bos@559: using the option to the qpush command. bos@580: cd inotify bos@580: hg qpush -a bos@559: bos@584: If you get an error message from qpush, you should not continue. bos@559: Instead, ask for help. bos@559: bos@584: Build and install the patched version of bos@559: Mercurial. bos@580: python setup.py build --force bos@580: sudo python setup.py install --skip-build bos@559: bos@559: bos@584: Once you've build a suitably patched version of Mercurial, bos@559: all you need to do to enable the inotify extension is add an entry to bos@580: your ~/.hgrc. bos@559: [extensions] inotify = bos@584: When the inotify extension bos@559: is enabled, Mercurial will automatically and transparently start bos@559: the status daemon the first time you run a command that needs bos@559: status in a repository. It runs one status daemon per bos@559: repository. bos@559: bos@584: The status daemon is started silently, and runs in the bos@559: background. If you look at a list of running processes after bos@559: you've enabled the inotify bos@559: extension and run a few commands in different repositories, bos@559: you'll thus see a few hg processes sitting bos@559: around, waiting for updates from the kernel and queries from bos@559: Mercurial. bos@559: bos@584: The first time you run a Mercurial command in a repository bos@559: when you have the inotify bos@559: extension enabled, it will run with about the same performance bos@559: as a normal Mercurial command. This is because the status bos@559: daemon needs to perform a normal status scan so that it has a bos@559: baseline against which to apply later updates from the kernel. bos@559: However, every subsequent command that does bos@559: any kind of status check should be noticeably faster on bos@559: repositories of even fairly modest size. Better yet, the bigger bos@559: your repository is, the greater a performance advantage you'll bos@559: see. The inotify daemon makes bos@559: status operations almost instantaneous on repositories of all bos@559: sizes! bos@559: bos@584: If you like, you can manually start a status daemon using bos@559: the inserve command. bos@559: This gives you slightly finer control over how the daemon ought bos@559: to run. This command will of course only be available when the bos@559: inotify extension is bos@559: enabled. bos@559: bos@584: When you're using the inotify extension, you should notice bos@559: no difference at all in Mercurial's bos@672: behavior, with the sole exception of status-related commands bos@559: running a whole lot faster than they used to. You should bos@559: specifically expect that commands will not print different bos@559: output; neither should they give different results. If either of bos@559: these situations occurs, please report a bug. bos@559: bos@559: bos@559: bos@559: Flexible diff support with the <literal bos@559: role="hg-ext">extdiff</literal> extension bos@559: bos@584: Mercurial's built-in hg bos@567: diff command outputs plaintext unified diffs. bos@567: bos@567: &interaction.extdiff.diff; bos@567: bos@584: If you would like to use an external tool to display bos@567: modifications, you'll want to use the extdiff extension. This will let you bos@567: use, for example, a graphical diff tool. bos@559: bos@584: The extdiff extension is bos@559: bundled with Mercurial, so it's easy to set up. In the extensions section of your bos@580: ~/.hgrc, simply add a bos@559: one-line entry to enable the extension. bos@580: [extensions] bos@580: extdiff = bos@584: This introduces a command named extdiff, which by default uses bos@559: your system's diff command to generate a bos@559: unified diff in the same form as the built-in hg diff command. bos@567: bos@567: &interaction.extdiff.extdiff; bos@567: bos@584: The result won't be exactly the same as with the built-in bos@567: hg diff variations, because the bos@567: output of diff varies from one system to bos@567: another, even when passed the same options. bos@559: bos@584: As the making snapshot bos@559: lines of output above imply, the extdiff command works by bos@559: creating two snapshots of your source tree. The first snapshot bos@559: is of the source revision; the second, of the target revision or bos@559: working directory. The extdiff command generates bos@559: these snapshots in a temporary directory, passes the name of bos@559: each directory to an external diff viewer, then deletes the bos@559: temporary directory. For efficiency, it only snapshots the bos@559: directories and files that have changed between the two bos@559: revisions. bos@559: bos@584: Snapshot directory names have the same base name as your bos@559: repository. If your repository path is /quux/bar/foo, then foo will be the name of each bos@559: snapshot directory. Each snapshot directory name has its bos@559: changeset ID appended, if appropriate. If a snapshot is of bos@559: revision a631aca1083f, the directory will be bos@559: named foo.a631aca1083f. bos@559: A snapshot of the working directory won't have a changeset ID bos@559: appended, so it would just be foo in this example. To see what bos@559: this looks like in practice, look again at the extdiff example above. Notice bos@559: that the diff has the snapshot directory names embedded in its bos@559: header. bos@559: bos@584: The extdiff command bos@559: accepts two important options. The option bos@559: lets you choose a program to view differences with, instead of bos@559: diff. With the option, bos@559: you can change the options that extdiff passes to the program bos@559: (by default, these options are bos@559: -Npru, which only make sense bos@559: if you're running diff). In other respects, bos@559: the extdiff command bos@559: acts similarly to the built-in hg bos@559: diff command: you use the same option names, syntax, bos@559: and arguments to specify the revisions you want, the files you bos@559: want, and so on. bos@559: bos@584: As an example, here's how to run the normal system bos@559: diff command, getting it to generate context bos@559: diffs (using the option) bos@559: instead of unified diffs, and five lines of context instead of bos@559: the default three (passing 5 as the argument bos@567: to the option). bos@567: bos@567: &interaction.extdiff.extdiff-ctx; bos@559: bos@584: Launching a visual diff tool is just as easy. Here's how to bos@559: launch the kdiff3 viewer. bos@559: hg extdiff -p kdiff3 -o bos@559: bos@584: If your diff viewing command can't deal with directories, bos@559: you can easily work around this with a little scripting. For an bos@559: example of such scripting in action with the mq extension and the bos@592: interdiff command, see . bos@559: bos@559: bos@559: Defining command aliases bos@559: bos@584: It can be cumbersome to remember the options to both the bos@559: extdiff command and bos@559: the diff viewer you want to use, so the extdiff extension lets you define bos@559: new commands that will invoke your diff bos@559: viewer with exactly the right options. bos@559: bos@584: All you need to do is edit your ~/.hgrc, and add a section named bos@580: extdiff. Inside this bos@580: section, you can define multiple commands. Here's how to add bos@580: a kdiff3 command. Once you've defined bos@580: this, you can type hg kdiff3 bos@580: and the extdiff extension bos@580: will run kdiff3 for you. bos@580: [extdiff] bos@580: cmd.kdiff3 = bos@584: If you leave the right hand side of the definition empty, bos@559: as above, the extdiff bos@559: extension uses the name of the command you defined as the name bos@559: of the external program to run. But these names don't have to bos@559: be the same. Here, we define a command named bos@559: hg wibble, which runs bos@559: kdiff3. bos@580: [extdiff] bos@580: cmd.wibble = kdiff3 bos@559: bos@584: You can also specify the default options that you want to bos@559: invoke your diff viewing program with. The prefix to use is bos@559: opts., followed by the name bos@559: of the command to which the options apply. This example bos@559: defines a hg vimdiff command bos@559: that runs the vim editor's bos@559: DirDiff extension. bos@580: [extdiff] bos@580: cmd.vimdiff = vim bos@580: opts.vimdiff = -f '+next' '+execute "DirDiff" argv(0) argv(1)' bos@559: bos@559: bos@559: bos@559: bos@559: Cherrypicking changes with the <literal bos@559: role="hg-ext">transplant</literal> extension bos@559: bos@584: Need to have a long chat with Brendan about this. bos@559: bos@559: bos@559: bos@559: Send changes via email with the <literal bos@559: role="hg-ext">patchbomb</literal> extension bos@559: bos@584: Many projects have a culture of change bos@559: review, in which people send their modifications to a bos@559: mailing list for others to read and comment on before they bos@559: commit the final version to a shared repository. Some projects bos@559: have people who act as gatekeepers; they apply changes from bos@559: other people to a repository to which those others don't have bos@559: access. bos@559: bos@584: Mercurial makes it easy to send changes over email for bos@559: review or application, via its patchbomb extension. The extension is ori@561: so named because changes are formatted as patches, and it's usual bos@559: to send one changeset per email message. Sending a long series bos@559: of changes by email is thus much like bombing the bos@559: recipient's inbox, hence patchbomb. bos@559: bos@584: As usual, the basic configuration of the patchbomb extension takes just one or bos@559: two lines in your bos@559: /.hgrc. bos@580: [extensions] bos@580: patchbomb = bos@584: Once you've enabled the extension, you will have a new bos@559: command available, named email. bos@559: bos@584: The safest and best way to invoke the email command is to bos@559: always run it first with the option. bos@559: This will show you what the command would bos@559: send, without actually sending anything. Once you've had a bos@559: quick glance over the changes and verified that you are sending bos@559: the right ones, you can rerun the same command, with the option bos@559: removed. bos@559: bos@584: The email command bos@559: accepts the same kind of revision syntax as every other bos@559: Mercurial command. For example, this command will send every bos@559: revision between 7 and tip, inclusive. bos@559: hg email -n 7:tip bos@584: You can also specify a repository to bos@559: compare with. If you provide a repository but no revisions, the bos@559: email command will bos@559: send all revisions in the local repository that are not present bos@559: in the remote repository. If you additionally specify revisions bos@559: or a branch name (the latter using the option), bos@559: this will constrain the revisions sent. bos@559: bos@584: It's perfectly safe to run the email command without the bos@559: names of the people you want to send to: if you do this, it will bos@559: just prompt you for those values interactively. (If you're bos@559: using a Linux or Unix-like system, you should have enhanced bos@559: readline-style editing capabilities when bos@559: entering those headers, too, which is useful.) bos@559: bos@584: When you are sending just one revision, the email command will by bos@559: default use the first line of the changeset description as the bos@559: subject of the single email message it sends. bos@559: bos@584: If you send multiple revisions, the email command will usually bos@559: send one message per changeset. It will preface the series with bos@559: an introductory message, in which you should describe the bos@559: purpose of the series of changes you're sending. bos@559: bos@559: bos@672: Changing the behavior of patchbombs bos@559: bos@584: Not every project has exactly the same conventions for bos@559: sending changes in email; the patchbomb extension tries to bos@559: accommodate a number of variations through command line bos@559: options. bos@559: bos@584: You can write a subject for the introductory bos@559: message on the command line using the bos@559: option. This takes one argument, the text of the subject bos@559: to use. bos@559: bos@584: To change the email address from which the bos@559: messages originate, use the bos@559: option. This takes one argument, the email address to bos@559: use. bos@559: bos@672: The default behavior is to send unified diffs bos@592: (see for a bos@559: description of the bos@559: format), one per message. You can send a binary bundle bos@559: instead with the bos@559: option. bos@559: bos@584: Unified diffs are normally prefaced with a bos@559: metadata header. You can omit this, and send unadorned bos@559: diffs, with the option. bos@559: bos@584: Diffs are normally sent inline, bos@559: in the same body part as the description of a patch. This bos@559: makes it easiest for the largest number of readers to bos@559: quote and respond to parts of a diff, as some mail clients bos@559: will only quote the first MIME body part in a message. If bos@559: you'd prefer to send the description and the diff in bos@559: separate body parts, use the bos@559: option. bos@559: bos@584: Instead of sending mail messages, you can bos@559: write them to an mbox-format mail bos@559: folder using the bos@559: option. That option takes one argument, the name of the bos@559: file to write to. bos@559: bos@584: If you would like to add a bos@559: diffstat-format summary to each patch, bos@559: and one to the introductory message, use the bos@559: option. The diffstat command displays bos@559: a table containing the name of each file patched, the bos@559: number of lines affected, and a histogram showing how much bos@559: each file is modified. This gives readers a qualitative bos@559: glance at how complex a patch is. bos@559: bos@559: bos@559: bos@559: bos@559: bos@559: bos@559: