bos@104: \chapter{Advanced uses of Mercurial Queues} bos@104: bos@104: While it's easy to pick up straightforward uses of Mercurial Queues, bos@104: use of a little discipline and some of MQ's less frequently used bos@104: capabilities makes it possible to work in complicated development bos@104: environments. bos@104: bos@105: In this chapter, I will use as an example a technique I have used to bos@105: manage the development of an Infiniband device driver for the Linux bos@105: kernel. The driver in question is large (at least as drivers go), bos@105: with 25,000 lines of code spread across 35 source files. It is bos@105: maintained by a small team of developers. bos@104: bos@104: While much of the material in this chapter is specific to Linux, the bos@104: same principles apply to any code base for which you're not the bos@104: primary owner, and upon which you need to do a lot of development. bos@104: bos@104: \section{The problem of many targets} bos@104: bos@104: The Linux kernel changes rapidly, and has never been internally bos@104: stable; developers frequently make drastic changes between releases. bos@104: This means that a version of the driver that works well with a bos@104: particular released version of the kernel will not even \emph{compile} bos@104: correctly against, typically, any other version. bos@104: bos@104: To maintain a driver, we have to keep a number of distinct versions of bos@104: Linux in mind. bos@104: \begin{itemize} bos@104: \item One target is the main Linux kernel development tree. bos@104: Maintenance of the code is in this case partly shared by other bos@104: developers in the kernel community, who make ``drive-by'' bos@104: modifications to the driver as they develop and refine kernel bos@104: subsystems. bos@104: \item We also maintain a number of ``backports'' to older versions of bos@104: the Linux kernel, to support the needs of customers who are running bos@105: older Linux distributions that do not incorporate our drivers. (To bos@105: \emph{backport} a piece of code is to modify it to work in an older bos@105: version of its target environment than the version it was developed bos@105: for.) bos@104: \item Finally, we make software releases on a schedule that is bos@104: necessarily not aligned with those used by Linux distributors and bos@104: kernel developers, so that we can deliver new features to customers bos@104: without forcing them to upgrade their entire kernels or bos@104: distributions. bos@104: \end{itemize} bos@104: bos@104: \subsection{Tempting approaches that don't work well} bos@104: bos@104: There are two ``standard'' ways to maintain a piece of software that bos@104: has to target many different environments. bos@104: bos@104: The first is to maintain a number of branches, each intended for a bos@104: single target. The trouble with this approach is that you must bos@104: maintain iron discipline in the flow of changes between repositories. bos@104: A new feature or bug fix must start life in a ``pristine'' repository, bos@104: then percolate out to every backport repository. Backport changes are bos@104: more limited in the branches they should propagate to; a backport bos@104: change that is applied to a branch where it doesn't belong will bos@104: probably stop the driver from compiling. bos@104: bos@104: The second is to maintain a single source tree filled with conditional bos@104: statements that turn chunks of code on or off depending on the bos@104: intended target. Because these ``ifdefs'' are not allowed in the bos@104: Linux kernel tree, a manual or automatic process must be followed to bos@104: strip them out and yield a clean tree. A code base maintained in this bos@104: fashion rapidly becomes a rat's nest of conditional blocks that are bos@104: difficult to understand and maintain. bos@104: bos@104: Neither of these approaches is well suited to a situation where you bos@104: don't ``own'' the canonical copy of a source tree. In the case of a bos@104: Linux driver that is distributed with the standard kernel, Linus's bos@104: tree contains the copy of the code that will be treated by the world bos@104: as canonical. The upstream version of ``my'' driver can be modified bos@104: by people I don't know, without me even finding out about it until bos@104: after the changes show up in Linus's tree. bos@104: bos@104: These approaches have the added weakness of making it difficult to bos@104: generate well-formed patches to submit upstream. bos@104: bos@104: In principle, Mercurial Queues seems like a good candidate to manage a bos@104: development scenario such as the above. While this is indeed the bos@104: case, MQ contains a few added features that make the job more bos@104: pleasant. bos@104: bos@105: \section{Conditionally applying patches with bos@105: guards} bos@104: bos@104: Perhaps the best way to maintain sanity with so many targets is to be bos@104: able to choose specific patches to apply for a given situation. MQ bos@104: provides a feature called ``guards'' (which originates with quilt's bos@104: \texttt{guards} command) that does just this. To start off, let's bos@104: create a simple repository for experimenting in. bos@104: \interaction{mq.guards.init} bos@104: This gives us a tiny repository that contains two patches that don't bos@104: have any dependencies on each other, because they touch different files. bos@104: bos@104: The idea behind conditional application is that you can ``tag'' a bos@104: patch with a \emph{guard}, which is simply a text string of your bos@104: choosing, then tell MQ to select specific guards to use when applying bos@104: patches. MQ will then either apply, or skip over, a guarded patch, bos@104: depending on the guards that you have selected. bos@104: bos@104: A patch can have an arbitrary number of guards; bos@104: each one is \emph{positive} (``apply this patch if this guard is bos@104: selected'') or \emph{negative} (``skip this patch if this guard is bos@104: selected''). A patch with no guards is always applied. bos@104: bos@104: \section{Controlling the guards on a patch} bos@104: bos@104: The \hgcmd{qguard} command lets you determine which guards should bos@104: apply to a patch, or display the guards that are already in effect. bos@104: Without any arguments, it displays the guards on the current topmost bos@104: patch. bos@104: \interaction{mq.guards.qguard} bos@104: To set a positive guard on a patch, prefix the name of the guard with bos@104: a ``\texttt{+}''. bos@104: \interaction{mq.guards.qguard.pos} bos@104: To set a negative guard on a patch, prefix the name of the guard with bos@104: a ``\texttt{-}''. bos@104: \interaction{mq.guards.qguard.neg} bos@104: bos@104: \begin{note} bos@104: The \hgcmd{qguard} command \emph{sets} the guards on a patch; it bos@104: doesn't \emph{modify} them. What this means is that if you run bos@104: \hgcmdargs{qguard}{+a +b} on a patch, then \hgcmdargs{qguard}{+c} on bos@104: the same patch, the \emph{only} guard that will be set on it bos@104: afterwards is \texttt{+c}. bos@104: \end{note} bos@104: bos@104: Mercurial stores guards in the \sfilename{series} file; the form in bos@104: which they are stored is easy both to understand and to edit by hand. bos@104: (In other words, you don't have to use the \hgcmd{qguard} command if bos@104: you don't want to; it's okay to simply edit the \sfilename{series} bos@104: file.) bos@104: \interaction{mq.guards.series} bos@104: bos@104: \section{Selecting the guards to use} bos@104: bos@104: The \hgcmd{qselect} command determines which guards are active at a bos@104: given time. The effect of this is to determine which patches MQ will bos@104: apply the next time you run \hgcmd{qpush}. It has no other effect; in bos@104: particular, it doesn't do anything to patches that are already bos@104: applied. bos@104: bos@104: With no arguments, the \hgcmd{qselect} command lists the guards bos@104: currently in effect, one per line of output. Each argument is treated bos@104: as the name of a guard to apply. bos@104: \interaction{mq.guards.qselect.foo} bos@104: In case you're interested, the currently selected guards are stored in bos@104: the \sfilename{guards} file. bos@104: \interaction{mq.guards.qselect.cat} bos@104: We can see the effect the selected guards have when we run bos@104: \hgcmd{qpush}. bos@104: \interaction{mq.guards.qselect.qpush} bos@104: bos@104: A guard cannot start with a ``\texttt{+}'' or ``\texttt{-}'' bos@105: character. The name of a guard must start with an alphabetic bos@105: character (upper or lower case) or an underscore. The rest of the bos@105: guard's name can contain any of these characters, or a digit. These bos@105: rules are similar to those used for variable naming in most popular bos@105: programming languages. If you try to use a guard with an invalid bos@105: name, MQ will complain: bos@104: \interaction{mq.guards.qselect.error} bos@104: Changing the selected guards changes the patches that are applied. bos@104: \interaction{mq.guards.qselect.quux} bos@105: You can see in the example below that negative guards take precedence bos@105: over positive guards. bos@104: \interaction{mq.guards.qselect.foobar} bos@104: bos@105: \section{MQ's rules for applying patches} bos@105: bos@105: The rules that MQ uses when deciding whether to apply a patch bos@105: are as follows. bos@105: \begin{itemize} bos@105: \item A patch that has no guards is always applied. bos@105: \item If the patch has any negative guard that matches any currently bos@105: selected guard, the patch is skipped. bos@105: \item If the patch has any positive guard that matches any currently bos@105: selected guard, the patch is applied. bos@105: \item If the patch has positive or negative guards, but none matches bos@105: any currently selected guard, the patch is skipped. bos@105: \end{itemize} bos@105: bos@105: \section{Trimming the work environment} bos@105: bos@105: In working on the device driver I mentioned earlier, I don't apply the bos@105: patches to a normal Linux kernel tree. Instead, I use a repository bos@105: that contains only a snapshot of the source files and headers that are bos@105: relevant to Infiniband development. This repository is~1\% the size bos@105: of a kernel repository, so it's easier to work with. bos@105: bos@105: I then choose a ``base'' version on top of which the patches are bos@105: applied. This is a snapshot of the Linux kernel tree as of a revision bos@105: of my choosing. When I take the snapshot, I record the changeset ID bos@105: from the kernel repository in the commit message. Since the snapshot bos@105: preserves the ``shape'' and content of the relevant parts of the bos@105: kernel tree, I can apply my patches on top of either my tiny bos@105: repository or a normal kernel tree. bos@105: bos@105: Normally, the base tree atop which the patches apply should be a bos@105: snapshot of a very recent upstream tree. This best facilitates the bos@105: development of patches that can easily be submitted upstream with few bos@105: or no modifications. bos@105: bos@105: \section{Dividing up the \sfilename{series} file} bos@105: bos@105: I categorise the patches in the \sfilename{series} file into a number bos@105: of logical groups. Each section of like patches begins with a block bos@105: of comments that describes the purpose of the patches that follow. bos@105: bos@105: The sequence of patch groups that I maintain follows. The ordering of bos@105: these groups is important; I'll describe why after I introduce the bos@105: groups. bos@105: \begin{itemize} bos@105: \item The ``accepted'' group. Patches that the development team has bos@105: submitted to the maintainer of the Infiniband subsystem, and which bos@105: he has accepted, but which are not present in the snapshot that the bos@105: tiny repository is based on. These are ``read only'' patches, bos@105: present only to transform the tree into a similar state as it is in bos@105: the upstream maintainer's repository. bos@105: \item The ``rework'' group. Patches that I have submitted, but that bos@105: the upstream maintainer has requested modifications to before he bos@105: will accept them. bos@105: \item The ``pending'' group. Patches that I have not yet submitted to bos@105: the upstream maintainer, but which we have finished working on. bos@105: These will be ``read only'' for a while. If the upstream maintainer bos@105: accepts them upon submission, I'll move them to the end of the bos@105: ``accepted'' group. If he requests that I modify any, I'll move bos@105: them to the beginning of the ``rework'' group. bos@105: \item The ``in progress'' group. Patches that are actively being bos@105: developed, and should not be submitted anywhere yet. bos@105: \item The ``backport'' group. Patches that adapt the source tree to bos@105: older versions of the kernel tree. bos@105: \item The ``do not ship'' group. Patches that for some reason should bos@105: never be submitted upstream. For example, one such patch might bos@105: change embedded driver identification strings to make it easier to bos@105: distinguish, in the field, between an out-of-tree version of the bos@105: driver and a version shipped by a distribution vendor. bos@105: \end{itemize} bos@105: bos@105: Now to return to the reasons for ordering groups of patches in this bos@105: way. We would like the lowest patches in the stack to be as stable as bos@105: possible, so that we will not need to rework higher patches due to bos@105: changes in context. Putting patches that will never be changed first bos@105: in the \sfilename{series} file serves this purpose. bos@105: bos@105: We would also like the patches that we know we'll need to modify to be bos@105: applied on top of a source tree that resembles the upstream tree as bos@105: closely as possible. This is why we keep accepted patches around for bos@105: a while. bos@105: bos@105: The ``backport'' and ``do not ship'' patches float at the end of the bos@105: \sfilename{series} file in part because they'll never be shipped bos@105: upstream. Additionally, the backport patches must be applied on top bos@105: of all other patches. bos@105: bos@104: %%% Local Variables: bos@104: %%% mode: latex bos@104: %%% TeX-master: "00book" bos@104: %%% End: