bos@559: bos@559: dongsheng@625: bos@572: bos@559: Advanced uses of Mercurial Queues bos@559: bos@559: While it's easy to pick up straightforward uses of Mercurial bos@559: Queues, use of a little discipline and some of MQ's less bos@559: frequently used capabilities makes it possible to work in bos@559: complicated development environments. bos@559: bos@559: In this chapter, I will use as an example a technique I have bos@559: used to manage the development of an Infiniband device driver for bos@559: the Linux kernel. The driver in question is large (at least as bos@559: drivers go), with 25,000 lines of code spread across 35 source bos@559: files. It is maintained by a small team of developers. bos@559: bos@559: While much of the material in this chapter is specific to bos@559: Linux, the same principles apply to any code base for which you're bos@559: not the primary owner, and upon which you need to do a lot of bos@559: development. bos@559: bos@559: bos@559: The problem of many targets bos@559: bos@559: The Linux kernel changes rapidly, and has never been bos@559: internally stable; developers frequently make drastic changes bos@559: between releases. This means that a version of the driver that bos@559: works well with a particular released version of the kernel will bos@559: not even compile correctly against, bos@559: typically, any other version. bos@559: bos@559: To maintain a driver, we have to keep a number of distinct bos@559: versions of Linux in mind. bos@559: bos@559: One target is the main Linux kernel development bos@559: tree. Maintenance of the code is in this case partly shared bos@559: by other developers in the kernel community, who make bos@559: drive-by modifications to the driver as they bos@559: develop and refine kernel subsystems. bos@559: bos@559: We also maintain a number of bos@559: backports to older versions of the Linux bos@559: kernel, to support the needs of customers who are running bos@559: older Linux distributions that do not incorporate our bos@559: drivers. (To backport a piece of code bos@559: is to modify it to work in an older version of its target bos@559: environment than the version it was developed for.) bos@559: bos@559: Finally, we make software releases on a schedule bos@559: that is necessarily not aligned with those used by Linux bos@559: distributors and kernel developers, so that we can deliver bos@559: new features to customers without forcing them to upgrade bos@559: their entire kernels or distributions. bos@559: bos@559: bos@559: bos@559: Tempting approaches that don't work well bos@559: bos@559: There are two standard ways to maintain a bos@559: piece of software that has to target many different bos@559: environments. bos@559: bos@559: The first is to maintain a number of branches, each bos@559: intended for a single target. The trouble with this approach bos@559: is that you must maintain iron discipline in the flow of bos@559: changes between repositories. A new feature or bug fix must bos@559: start life in a pristine repository, then bos@559: percolate out to every backport repository. Backport changes bos@559: are more limited in the branches they should propagate to; a bos@559: backport change that is applied to a branch where it doesn't bos@559: belong will probably stop the driver from compiling. bos@559: bos@559: The second is to maintain a single source tree filled with bos@559: conditional statements that turn chunks of code on or off bos@559: depending on the intended target. Because these bos@559: ifdefs are not allowed in the Linux kernel bos@559: tree, a manual or automatic process must be followed to strip bos@559: them out and yield a clean tree. A code base maintained in bos@559: this fashion rapidly becomes a rat's nest of conditional bos@559: blocks that are difficult to understand and maintain. bos@559: bos@559: Neither of these approaches is well suited to a situation bos@559: where you don't own the canonical copy of a bos@559: source tree. In the case of a Linux driver that is bos@559: distributed with the standard kernel, Linus's tree contains bos@559: the copy of the code that will be treated by the world as bos@559: canonical. The upstream version of my driver bos@559: can be modified by people I don't know, without me even bos@559: finding out about it until after the changes show up in bos@559: Linus's tree. bos@559: bos@559: These approaches have the added weakness of making it bos@559: difficult to generate well-formed patches to submit bos@559: upstream. bos@559: bos@559: In principle, Mercurial Queues seems like a good candidate bos@559: to manage a development scenario such as the above. While bos@559: this is indeed the case, MQ contains a few added features that bos@559: make the job more pleasant. bos@559: bos@559: bos@559: bos@559: bos@559: Conditionally applying patches with guards bos@559: bos@559: Perhaps the best way to maintain sanity with so many targets bos@559: is to be able to choose specific patches to apply for a given bos@559: situation. MQ provides a feature called guards bos@559: (which originates with quilt's guards bos@559: command) that does just this. To start off, let's create a bos@567: simple repository for experimenting in. bos@567: bos@567: &interaction.mq.guards.init; bos@567: bos@567: This gives us a tiny repository that contains two patches bos@567: that don't have any dependencies on each other, because they bos@567: touch different files. bos@559: bos@559: The idea behind conditional application is that you can bos@559: tag a patch with a guard, bos@559: which is simply a text string of your choosing, then tell MQ to bos@559: select specific guards to use when applying patches. MQ will bos@559: then either apply, or skip over, a guarded patch, depending on bos@559: the guards that you have selected. bos@559: bos@559: A patch can have an arbitrary number of guards; each one is bos@559: positive (apply this patch if this bos@559: guard is selected) or negative bos@559: (skip this patch if this guard is selected). A bos@559: patch with no guards is always applied. bos@559: bos@559: bos@559: bos@559: Controlling the guards on a patch bos@559: bos@559: The qguard command lets bos@559: you determine which guards should apply to a patch, or display bos@559: the guards that are already in effect. Without any arguments, it bos@567: displays the guards on the current topmost patch. bos@567: bos@567: &interaction.mq.guards.qguard; bos@567: bos@567: To set a positive guard on a patch, prefix the name of the bos@567: guard with a +. bos@567: bos@567: &interaction.mq.guards.qguard.pos; bos@567: bos@567: To set a negative guard bos@559: on a patch, prefix the name of the guard with a bos@567: -. bos@567: bos@567: &interaction.mq.guards.qguard.neg; bos@559: bos@559: bos@559: The qguard command bos@559: sets the guards on a patch; it doesn't bos@559: modify them. What this means is that if bos@559: you run hg qguard +a +b on a bos@559: patch, then hg qguard +c on bos@559: the same patch, the only guard that will bos@559: be set on it afterwards is +c. bos@559: bos@559: bos@559: Mercurial stores guards in the series file; the form in which they bos@559: are stored is easy both to understand and to edit by hand. (In bos@559: other words, you don't have to use the qguard command if you don't want bos@559: to; it's okay to simply edit the series file.) bos@567: bos@567: &interaction.mq.guards.series; bos@559: bos@559: bos@559: bos@559: Selecting the guards to use bos@559: bos@559: The qselect command bos@559: determines which guards are active at a given time. The effect bos@559: of this is to determine which patches MQ will apply the next bos@559: time you run qpush. It has bos@559: no other effect; in particular, it doesn't do anything to bos@559: patches that are already applied. bos@559: bos@559: With no arguments, the qselect command lists the guards bos@559: currently in effect, one per line of output. Each argument is bos@567: treated as the name of a guard to apply. bos@567: bos@567: &interaction.mq.guards.qselect.foo; bos@567: bos@567: In case you're interested, the currently selected guards are bos@567: stored in the guards file. bos@567: bos@567: &interaction.mq.guards.qselect.cat; bos@567: bos@567: We can see the effect the selected guards have when we run bos@567: qpush. bos@567: bos@567: &interaction.mq.guards.qselect.qpush; bos@559: bos@559: A guard cannot start with a bos@559: + or bos@559: - character. The name of a bos@559: guard must not contain white space, but most other characters bos@559: are acceptable. If you try to use a guard with an invalid name, bos@567: MQ will complain: bos@567: bos@567: &interaction.mq.guards.qselect.error; bos@567: bos@567: Changing the selected guards changes the patches that are bos@567: applied. bos@567: bos@567: &interaction.mq.guards.qselect.quux; bos@567: bos@567: You can see in the example below that negative guards take bos@567: precedence over positive guards. bos@567: bos@567: &interaction.mq.guards.qselect.foobar; bos@559: bos@559: bos@559: bos@559: MQ's rules for applying patches bos@559: bos@559: The rules that MQ uses when deciding whether to apply a bos@559: patch are as follows. bos@559: bos@559: A patch that has no guards is always bos@559: applied. bos@559: bos@559: If the patch has any negative guard that matches bos@559: any currently selected guard, the patch is skipped. bos@559: bos@559: If the patch has any positive guard that matches bos@559: any currently selected guard, the patch is applied. bos@559: bos@559: If the patch has positive or negative guards, bos@559: but none matches any currently selected guard, the patch is bos@559: skipped. bos@559: bos@559: bos@559: bos@559: bos@559: Trimming the work environment bos@559: bos@559: In working on the device driver I mentioned earlier, I don't bos@559: apply the patches to a normal Linux kernel tree. Instead, I use bos@559: a repository that contains only a snapshot of the source files bos@559: and headers that are relevant to Infiniband development. This bos@559: repository is 1% the size of a kernel repository, so it's easier bos@559: to work with. bos@559: bos@559: I then choose a base version on top of which bos@559: the patches are applied. This is a snapshot of the Linux kernel bos@559: tree as of a revision of my choosing. When I take the snapshot, bos@559: I record the changeset ID from the kernel repository in the bos@559: commit message. Since the snapshot preserves the bos@559: shape and content of the relevant parts of the bos@559: kernel tree, I can apply my patches on top of either my tiny bos@559: repository or a normal kernel tree. bos@559: bos@559: Normally, the base tree atop which the patches apply should bos@559: be a snapshot of a very recent upstream tree. This best bos@559: facilitates the development of patches that can easily be bos@559: submitted upstream with few or no modifications. bos@559: bos@559: bos@559: bos@559: Dividing up the <filename role="special">series</filename> bos@559: file bos@559: bos@559: I categorise the patches in the series file into a number of logical bos@559: groups. Each section of like patches begins with a block of bos@559: comments that describes the purpose of the patches that bos@559: follow. bos@559: bos@559: The sequence of patch groups that I maintain follows. The bos@559: ordering of these groups is important; I'll describe why after I bos@559: introduce the groups. bos@559: bos@559: The accepted group. Patches that bos@559: the development team has submitted to the maintainer of the bos@559: Infiniband subsystem, and which he has accepted, but which bos@559: are not present in the snapshot that the tiny repository is bos@559: based on. These are read only patches, bos@559: present only to transform the tree into a similar state as bos@559: it is in the upstream maintainer's repository. bos@559: bos@559: The rework group. Patches that I bos@559: have submitted, but that the upstream maintainer has bos@559: requested modifications to before he will accept bos@559: them. bos@559: bos@559: The pending group. Patches that bos@559: I have not yet submitted to the upstream maintainer, but bos@559: which we have finished working on. These will be read bos@559: only for a while. If the upstream maintainer bos@559: accepts them upon submission, I'll move them to the end of bos@559: the accepted group. If he requests that I bos@559: modify any, I'll move them to the beginning of the bos@559: rework group. bos@559: bos@559: The in progress group. Patches bos@559: that are actively being developed, and should not be bos@559: submitted anywhere yet. bos@559: bos@559: The backport group. Patches that bos@559: adapt the source tree to older versions of the kernel bos@559: tree. bos@559: bos@559: The do not ship group. Patches bos@559: that for some reason should never be submitted upstream. bos@559: For example, one such patch might change embedded driver bos@559: identification strings to make it easier to distinguish, in bos@559: the field, between an out-of-tree version of the driver and bos@559: a version shipped by a distribution vendor. bos@559: bos@559: bos@559: Now to return to the reasons for ordering groups of patches bos@559: in this way. We would like the lowest patches in the stack to bos@559: be as stable as possible, so that we will not need to rework bos@559: higher patches due to changes in context. Putting patches that bos@559: will never be changed first in the series file serves this bos@559: purpose. bos@559: bos@559: We would also like the patches that we know we'll need to bos@559: modify to be applied on top of a source tree that resembles the bos@559: upstream tree as closely as possible. This is why we keep bos@559: accepted patches around for a while. bos@559: bos@559: The backport and do not ship bos@559: patches float at the end of the series file. The backport patches bos@559: must be applied on top of all other patches, and the do bos@559: not ship patches might as well stay out of harm's bos@559: way. bos@559: bos@559: bos@559: bos@559: Maintaining the patch series bos@559: bos@559: In my work, I use a number of guards to control which bos@559: patches are to be applied. bos@559: bos@559: bos@559: Accepted patches are guarded with bos@559: accepted. I enable this guard most of bos@559: the time. When I'm applying the patches on top of a tree bos@559: where the patches are already present, I can turn this patch bos@559: off, and the patches that follow it will apply bos@559: cleanly. bos@559: bos@559: Patches that are finished, but bos@559: not yet submitted, have no guards. If I'm applying the bos@559: patch stack to a copy of the upstream tree, I don't need to bos@559: enable any guards in order to get a reasonably safe source bos@559: tree. bos@559: bos@559: Those patches that need reworking before being bos@559: resubmitted are guarded with bos@559: rework. bos@559: bos@559: For those patches that are still under bos@559: development, I use devel. bos@559: bos@559: A backport patch may have several guards, one bos@559: for each version of the kernel to which it applies. For bos@559: example, a patch that backports a piece of code to 2.6.9 bos@559: will have a 2.6.9 guard. bos@559: bos@559: This variety of guards gives me considerable flexibility in bos@559: determining what kind of source tree I want to end up with. For bos@559: most situations, the selection of appropriate guards is bos@559: automated during the build process, but I can manually tune the bos@559: guards to use for less common circumstances. bos@559: bos@559: bos@559: The art of writing backport patches bos@559: bos@559: Using MQ, writing a backport patch is a simple process. bos@559: All such a patch has to do is modify a piece of code that uses bos@559: a kernel feature not present in the older version of the bos@559: kernel, so that the driver continues to work correctly under bos@559: that older version. bos@559: bos@559: A useful goal when writing a good backport patch is to bos@559: make your code look as if it was written for the older version bos@559: of the kernel you're targeting. The less obtrusive the patch, bos@559: the easier it will be to understand and maintain. If you're bos@559: writing a collection of backport patches to avoid the bos@559: rat's nest effect of lots of bos@559: #ifdefs (hunks of source code that are only bos@559: used conditionally) in your code, don't introduce bos@559: version-dependent #ifdefs into the patches. bos@559: Instead, write several patches, each of which makes bos@559: unconditional changes, and control their application using bos@559: guards. bos@559: bos@559: There are two reasons to divide backport patches into a bos@559: distinct group, away from the regular patches bos@559: whose effects they modify. The first is that intermingling the bos@559: two makes it more difficult to use a tool like the patchbomb extension to automate the bos@559: process of submitting the patches to an upstream maintainer. bos@559: The second is that a backport patch could perturb the context bos@559: in which a subsequent regular patch is applied, making it bos@559: impossible to apply the regular patch cleanly bos@559: without the earlier backport patch bos@559: already being applied. bos@559: bos@559: bos@559: bos@559: bos@559: Useful tips for developing with MQ bos@559: bos@559: bos@559: Organising patches in directories bos@559: bos@559: If you're working on a substantial project with MQ, it's bos@559: not difficult to accumulate a large number of patches. For bos@559: example, I have one patch repository that contains over 250 bos@559: patches. bos@559: bos@559: If you can group these patches into separate logical bos@559: categories, you can if you like store them in different bos@559: directories; MQ has no problems with patch names that contain bos@559: path separators. bos@559: bos@559: dongsheng@625: bos@559: Viewing the history of a patch bos@559: bos@559: If you're developing a set of patches over a long time, bos@559: it's a good idea to maintain them in a repository, as dongsheng@625: discussed in section . If you do bos@559: so, you'll quickly bos@559: discover that using the hg bos@559: diff command to look at the history of changes to bos@559: a patch is unworkable. This is in part because you're looking bos@559: at the second derivative of the real code (a diff of a diff), bos@559: but also because MQ adds noise to the process by modifying bos@559: time stamps and directory names when it updates a bos@559: patch. bos@559: bos@559: However, you can use the extdiff extension, which is bundled bos@559: with Mercurial, to turn a diff of two versions of a patch into bos@559: something readable. To do this, you will need a third-party bos@559: package called patchutils bos@559: web:patchutils. This provides a command bos@559: named interdiff, which shows the bos@559: differences between two diffs as a diff. Used on two versions bos@559: of the same diff, it generates a diff that represents the diff bos@559: from the first to the second version. bos@559: bos@559: You can enable the extdiff extension in the usual way, bos@559: by adding a line to the extensions section of your bos@580: ~/.hgrc. bos@580: [extensions] bos@580: extdiff = bos@559: The interdiff command expects to be bos@559: passed the names of two files, but the extdiff extension passes the program bos@559: it runs a pair of directories, each of which can contain an bos@559: arbitrary number of files. We thus need a small program that bos@559: will run interdiff on each pair of files in bos@559: these two directories. This program is available as hg-interdiff in the examples directory of the bos@559: source code repository that accompanies this book. bos@559: bos@559: With the hg-interdiff bos@559: program in your shell's search path, you can run it as bos@559: follows, from inside an MQ patch directory: bos@580: hg extdiff -p hg-interdiff -r A:B my-change.patch bos@559: Since you'll probably want to use this long-winded command bos@559: a lot, you can get hgext to bos@559: make it available as a normal Mercurial command, again by bos@580: editing your ~/.hgrc. bos@580: [extdiff] bos@580: cmd.interdiff = hg-interdiff bos@559: This directs hgext to bos@559: make an interdiff command available, so you bos@559: can now shorten the previous invocation of extdiff to something a bos@559: little more wieldy. bos@580: hg interdiff -r A:B my-change.patch bos@559: bos@559: bos@559: The interdiff command works well bos@559: only if the underlying files against which versions of a bos@559: patch are generated remain the same. If you create a patch, bos@559: modify the underlying files, and then regenerate the patch, bos@559: interdiff may not produce useful bos@559: output. bos@559: bos@559: bos@559: The extdiff extension is bos@559: useful for more than merely improving the presentation of MQ bos@559: patches. To read more about it, go to section . bos@559: bos@559: bos@559: bos@559: bos@559: bos@559: