hgbook
view fr/ch13-mq-collab.xml @ 981:64393e8da2ff
Some errors corrected in fr/ch00-preface.xml
| author | Frédéric Bouquet <youshe.jaalon@gmail.com> |
|---|---|
| date | Tue Sep 08 15:30:26 2009 +0200 (2009-09-08) |
| parents | |
| children | 6f8c48362758 |
line source
1 <!-- vim: set filetype=docbkxml shiftwidth=2 autoindent expandtab tw=77 : -->
3 <chapter>
4 <title>Advanced uses of Mercurial Queues</title>
5 <para>\label{chap:mq-collab}</para>
7 <para>While it's easy to pick up straightforward uses of Mercurial Queues,
8 use of a little discipline and some of MQ's less frequently used
9 capabilities makes it possible to work in complicated development
10 environments.</para>
12 <para>In this chapter, I will use as an example a technique I have used to
13 manage the development of an Infiniband device driver for the Linux
14 kernel. The driver in question is large (at least as drivers go),
15 with 25,000 lines of code spread across 35 source files. It is
16 maintained by a small team of developers.</para>
18 <para>While much of the material in this chapter is specific to Linux, the
19 same principles apply to any code base for which you're not the
20 primary owner, and upon which you need to do a lot of development.</para>
22 <sect1>
23 <title>The problem of many targets</title>
25 <para>The Linux kernel changes rapidly, and has never been internally
26 stable; developers frequently make drastic changes between releases.
27 This means that a version of the driver that works well with a
28 particular released version of the kernel will not even <emphasis>compile</emphasis>
29 correctly against, typically, any other version.</para>
31 <para>To maintain a driver, we have to keep a number of distinct versions of
32 Linux in mind.</para>
33 <itemizedlist>
34 <listitem><para>One target is the main Linux kernel development tree.
35 Maintenance of the code is in this case partly shared by other
36 developers in the kernel community, who make <quote>drive-by</quote>
37 modifications to the driver as they develop and refine kernel
38 subsystems.</para>
39 </listitem>
40 <listitem><para>We also maintain a number of <quote>backports</quote> to older versions of
41 the Linux kernel, to support the needs of customers who are running
42 older Linux distributions that do not incorporate our drivers. (To
43 <emphasis>backport</emphasis> a piece of code is to modify it to work in an older
44 version of its target environment than the version it was developed
45 for.)</para>
46 </listitem>
47 <listitem><para>Finally, we make software releases on a schedule that is
48 necessarily not aligned with those used by Linux distributors and
49 kernel developers, so that we can deliver new features to customers
50 without forcing them to upgrade their entire kernels or
51 distributions.
52 </para>
53 </listitem></itemizedlist>
55 <sect2>
56 <title>Tempting approaches that don't work well</title>
58 <para>There are two <quote>standard</quote> ways to maintain a piece of software that
59 has to target many different environments.
60 </para>
62 <para>The first is to maintain a number of branches, each intended for a
63 single target. The trouble with this approach is that you must
64 maintain iron discipline in the flow of changes between repositories.
65 A new feature or bug fix must start life in a <quote>pristine</quote> repository,
66 then percolate out to every backport repository. Backport changes are
67 more limited in the branches they should propagate to; a backport
68 change that is applied to a branch where it doesn't belong will
69 probably stop the driver from compiling.
70 </para>
72 <para>The second is to maintain a single source tree filled with conditional
73 statements that turn chunks of code on or off depending on the
74 intended target. Because these <quote>ifdefs</quote> are not allowed in the
75 Linux kernel tree, a manual or automatic process must be followed to
76 strip them out and yield a clean tree. A code base maintained in this
77 fashion rapidly becomes a rat's nest of conditional blocks that are
78 difficult to understand and maintain.
79 </para>
81 <para>Neither of these approaches is well suited to a situation where you
82 don't <quote>own</quote> the canonical copy of a source tree. In the case of a
83 Linux driver that is distributed with the standard kernel, Linus's
84 tree contains the copy of the code that will be treated by the world
85 as canonical. The upstream version of <quote>my</quote> driver can be modified
86 by people I don't know, without me even finding out about it until
87 after the changes show up in Linus's tree.
88 </para>
90 <para>These approaches have the added weakness of making it difficult to
91 generate well-formed patches to submit upstream.
92 </para>
94 <para>In principle, Mercurial Queues seems like a good candidate to manage a
95 development scenario such as the above. While this is indeed the
96 case, MQ contains a few added features that make the job more
97 pleasant.
98 </para>
100 <para>\section{Conditionally applying patches with
101 guards}
102 </para>
104 <para>Perhaps the best way to maintain sanity with so many targets is to be
105 able to choose specific patches to apply for a given situation. MQ
106 provides a feature called <quote>guards</quote> (which originates with quilt's
107 <literal>guards</literal> command) that does just this. To start off, let's
108 create a simple repository for experimenting in.
109 <!-- &interaction.mq.guards.init; -->
110 This gives us a tiny repository that contains two patches that don't
111 have any dependencies on each other, because they touch different files.
112 </para>
114 <para>The idea behind conditional application is that you can <quote>tag</quote> a
115 patch with a <emphasis>guard</emphasis>, which is simply a text string of your
116 choosing, then tell MQ to select specific guards to use when applying
117 patches. MQ will then either apply, or skip over, a guarded patch,
118 depending on the guards that you have selected.
119 </para>
121 <para>A patch can have an arbitrary number of guards;
122 each one is <emphasis>positive</emphasis> (<quote>apply this patch if this guard is
123 selected</quote>) or <emphasis>negative</emphasis> (<quote>skip this patch if this guard is
124 selected</quote>). A patch with no guards is always applied.
125 </para>
127 </sect2>
128 </sect1>
129 <sect1>
130 <title>Controlling the guards on a patch</title>
132 <para>The <command role="hg-ext-mq">qguard</command> command lets you determine which guards should
133 apply to a patch, or display the guards that are already in effect.
134 Without any arguments, it displays the guards on the current topmost
135 patch.
136 <!-- &interaction.mq.guards.qguard; -->
137 To set a positive guard on a patch, prefix the name of the guard with
138 a <quote><literal>+</literal></quote>.
139 <!-- &interaction.mq.guards.qguard.pos; -->
140 To set a negative guard on a patch, prefix the name of the guard with
141 a <quote><literal>-</literal></quote>.
142 <!-- &interaction.mq.guards.qguard.neg; -->
143 </para>
145 <note>
146 <para> The <command role="hg-ext-mq">qguard</command> command <emphasis>sets</emphasis> the guards on a patch; it
147 doesn't <emphasis>modify</emphasis> them. What this means is that if you run
148 <command role="hg-cmd">hg qguard +a +b</command> on a patch, then <command role="hg-cmd">hg qguard +c</command> on
149 the same patch, the <emphasis>only</emphasis> guard that will be set on it
150 afterwards is <literal>+c</literal>.
151 </para>
152 </note>
154 <para>Mercurial stores guards in the <filename role="special">series</filename> file; the form in
155 which they are stored is easy both to understand and to edit by hand.
156 (In other words, you don't have to use the <command role="hg-ext-mq">qguard</command> command if
157 you don't want to; it's okay to simply edit the <filename role="special">series</filename>
158 file.)
159 <!-- &interaction.mq.guards.series; -->
160 </para>
162 </sect1>
163 <sect1>
164 <title>Selecting the guards to use</title>
166 <para>The <command role="hg-ext-mq">qselect</command> command determines which guards are active at a
167 given time. The effect of this is to determine which patches MQ will
168 apply the next time you run <command role="hg-ext-mq">qpush</command>. It has no other effect; in
169 particular, it doesn't do anything to patches that are already
170 applied.
171 </para>
173 <para>With no arguments, the <command role="hg-ext-mq">qselect</command> command lists the guards
174 currently in effect, one per line of output. Each argument is treated
175 as the name of a guard to apply.
176 <!-- &interaction.mq.guards.qselect.foo; -->
177 In case you're interested, the currently selected guards are stored in
178 the <filename role="special">guards</filename> file.
179 <!-- &interaction.mq.guards.qselect.cat; -->
180 We can see the effect the selected guards have when we run
181 <command role="hg-ext-mq">qpush</command>.
182 <!-- &interaction.mq.guards.qselect.qpush; -->
183 </para>
185 <para>A guard cannot start with a <quote><literal>+</literal></quote> or <quote><literal>-</literal></quote>
186 character. The name of a guard must not contain white space, but most
187 other characters are acceptable. If you try to use a guard with an
188 invalid name, MQ will complain:
189 <!-- &interaction.mq.guards.qselect.error; -->
190 Changing the selected guards changes the patches that are applied.
191 <!-- &interaction.mq.guards.qselect.quux; -->
192 You can see in the example below that negative guards take precedence
193 over positive guards.
194 <!-- &interaction.mq.guards.qselect.foobar; -->
195 </para>
197 </sect1>
198 <sect1>
199 <title>MQ's rules for applying patches</title>
201 <para>The rules that MQ uses when deciding whether to apply a patch
202 are as follows.
203 </para>
204 <itemizedlist>
205 <listitem><para>A patch that has no guards is always applied.
206 </para>
207 </listitem>
208 <listitem><para>If the patch has any negative guard that matches any currently
209 selected guard, the patch is skipped.
210 </para>
211 </listitem>
212 <listitem><para>If the patch has any positive guard that matches any currently
213 selected guard, the patch is applied.
214 </para>
215 </listitem>
216 <listitem><para>If the patch has positive or negative guards, but none matches
217 any currently selected guard, the patch is skipped.
218 </para>
219 </listitem></itemizedlist>
221 </sect1>
222 <sect1>
223 <title>Trimming the work environment</title>
225 <para>In working on the device driver I mentioned earlier, I don't apply the
226 patches to a normal Linux kernel tree. Instead, I use a repository
227 that contains only a snapshot of the source files and headers that are
228 relevant to Infiniband development. This repository is 1% the size
229 of a kernel repository, so it's easier to work with.
230 </para>
232 <para>I then choose a <quote>base</quote> version on top of which the patches are
233 applied. This is a snapshot of the Linux kernel tree as of a revision
234 of my choosing. When I take the snapshot, I record the changeset ID
235 from the kernel repository in the commit message. Since the snapshot
236 preserves the <quote>shape</quote> and content of the relevant parts of the
237 kernel tree, I can apply my patches on top of either my tiny
238 repository or a normal kernel tree.
239 </para>
241 <para>Normally, the base tree atop which the patches apply should be a
242 snapshot of a very recent upstream tree. This best facilitates the
243 development of patches that can easily be submitted upstream with few
244 or no modifications.
245 </para>
247 </sect1>
248 <sect1>
249 <title>Dividing up the <filename role="special">series</filename> file</title>
251 <para>I categorise the patches in the <filename role="special">series</filename> file into a number
252 of logical groups. Each section of like patches begins with a block
253 of comments that describes the purpose of the patches that follow.
254 </para>
256 <para>The sequence of patch groups that I maintain follows. The ordering of
257 these groups is important; I'll describe why after I introduce the
258 groups.
259 </para>
260 <itemizedlist>
261 <listitem><para>The <quote>accepted</quote> group. Patches that the development team has
262 submitted to the maintainer of the Infiniband subsystem, and which
263 he has accepted, but which are not present in the snapshot that the
264 tiny repository is based on. These are <quote>read only</quote> patches,
265 present only to transform the tree into a similar state as it is in
266 the upstream maintainer's repository.
267 </para>
268 </listitem>
269 <listitem><para>The <quote>rework</quote> group. Patches that I have submitted, but that
270 the upstream maintainer has requested modifications to before he
271 will accept them.
272 </para>
273 </listitem>
274 <listitem><para>The <quote>pending</quote> group. Patches that I have not yet submitted to
275 the upstream maintainer, but which we have finished working on.
276 These will be <quote>read only</quote> for a while. If the upstream maintainer
277 accepts them upon submission, I'll move them to the end of the
278 <quote>accepted</quote> group. If he requests that I modify any, I'll move
279 them to the beginning of the <quote>rework</quote> group.
280 </para>
281 </listitem>
282 <listitem><para>The <quote>in progress</quote> group. Patches that are actively being
283 developed, and should not be submitted anywhere yet.
284 </para>
285 </listitem>
286 <listitem><para>The <quote>backport</quote> group. Patches that adapt the source tree to
287 older versions of the kernel tree.
288 </para>
289 </listitem>
290 <listitem><para>The <quote>do not ship</quote> group. Patches that for some reason should
291 never be submitted upstream. For example, one such patch might
292 change embedded driver identification strings to make it easier to
293 distinguish, in the field, between an out-of-tree version of the
294 driver and a version shipped by a distribution vendor.
295 </para>
296 </listitem></itemizedlist>
298 <para>Now to return to the reasons for ordering groups of patches in this
299 way. We would like the lowest patches in the stack to be as stable as
300 possible, so that we will not need to rework higher patches due to
301 changes in context. Putting patches that will never be changed first
302 in the <filename role="special">series</filename> file serves this purpose.
303 </para>
305 <para>We would also like the patches that we know we'll need to modify to be
306 applied on top of a source tree that resembles the upstream tree as
307 closely as possible. This is why we keep accepted patches around for
308 a while.
309 </para>
311 <para>The <quote>backport</quote> and <quote>do not ship</quote> patches float at the end of the
312 <filename role="special">series</filename> file. The backport patches must be applied on top
313 of all other patches, and the <quote>do not ship</quote> patches might as well
314 stay out of harm's way.
315 </para>
317 </sect1>
318 <sect1>
319 <title>Maintaining the patch series</title>
321 <para>In my work, I use a number of guards to control which patches are to
322 be applied.
323 </para>
325 <itemizedlist>
326 <listitem><para><quote>Accepted</quote> patches are guarded with <literal>accepted</literal>. I
327 enable this guard most of the time. When I'm applying the patches
328 on top of a tree where the patches are already present, I can turn
329 this patch off, and the patches that follow it will apply cleanly.
330 </para>
331 </listitem>
332 <listitem><para>Patches that are <quote>finished</quote>, but not yet submitted, have no
333 guards. If I'm applying the patch stack to a copy of the upstream
334 tree, I don't need to enable any guards in order to get a reasonably
335 safe source tree.
336 </para>
337 </listitem>
338 <listitem><para>Those patches that need reworking before being resubmitted are
339 guarded with <literal>rework</literal>.
340 </para>
341 </listitem>
342 <listitem><para>For those patches that are still under development, I use
343 <literal>devel</literal>.
344 </para>
345 </listitem>
346 <listitem><para>A backport patch may have several guards, one for each version
347 of the kernel to which it applies. For example, a patch that
348 backports a piece of code to 2.6.9 will have a <literal>2.6.9</literal> guard.
349 </para>
350 </listitem></itemizedlist>
351 <para>This variety of guards gives me considerable flexibility in
352 determining what kind of source tree I want to end up with. For most
353 situations, the selection of appropriate guards is automated during
354 the build process, but I can manually tune the guards to use for less
355 common circumstances.
356 </para>
358 <sect2>
359 <title>The art of writing backport patches</title>
361 <para>Using MQ, writing a backport patch is a simple process. All such a
362 patch has to do is modify a piece of code that uses a kernel feature
363 not present in the older version of the kernel, so that the driver
364 continues to work correctly under that older version.
365 </para>
367 <para>A useful goal when writing a good backport patch is to make your code
368 look as if it was written for the older version of the kernel you're
369 targeting. The less obtrusive the patch, the easier it will be to
370 understand and maintain. If you're writing a collection of backport
371 patches to avoid the <quote>rat's nest</quote> effect of lots of
372 <literal>#ifdef</literal>s (hunks of source code that are only used
373 conditionally) in your code, don't introduce version-dependent
374 <literal>#ifdef</literal>s into the patches. Instead, write several patches,
375 each of which makes unconditional changes, and control their
376 application using guards.
377 </para>
379 <para>There are two reasons to divide backport patches into a distinct
380 group, away from the <quote>regular</quote> patches whose effects they modify.
381 The first is that intermingling the two makes it more difficult to use
382 a tool like the <literal role="hg-ext">patchbomb</literal> extension to automate the process of
383 submitting the patches to an upstream maintainer. The second is that
384 a backport patch could perturb the context in which a subsequent
385 regular patch is applied, making it impossible to apply the regular
386 patch cleanly <emphasis>without</emphasis> the earlier backport patch already being
387 applied.
388 </para>
390 </sect2>
391 </sect1>
392 <sect1>
393 <title>Useful tips for developing with MQ</title>
395 <sect2>
396 <title>Organising patches in directories</title>
398 <para>If you're working on a substantial project with MQ, it's not difficult
399 to accumulate a large number of patches. For example, I have one
400 patch repository that contains over 250 patches.
401 </para>
403 <para>If you can group these patches into separate logical categories, you
404 can if you like store them in different directories; MQ has no
405 problems with patch names that contain path separators.
406 </para>
408 </sect2>
409 <sect2>
410 <title>Viewing the history of a patch</title>
411 <para>\label{mq-collab:tips:interdiff}
412 </para>
414 <para>If you're developing a set of patches over a long time, it's a good
415 idea to maintain them in a repository, as discussed in
416 section <xref linkend="sec:mq:repo"/>. If you do so, you'll quickly discover that
417 using the <command role="hg-cmd">hg diff</command> command to look at the history of changes to a
418 patch is unworkable. This is in part because you're looking at the
419 second derivative of the real code (a diff of a diff), but also
420 because MQ adds noise to the process by modifying time stamps and
421 directory names when it updates a patch.
422 </para>
424 <para>However, you can use the <literal role="hg-ext">extdiff</literal> extension, which is bundled
425 with Mercurial, to turn a diff of two versions of a patch into
426 something readable. To do this, you will need a third-party package
427 called <literal role="package">patchutils</literal> <citation>web:patchutils</citation>. This provides a
428 command named <command>interdiff</command>, which shows the differences between
429 two diffs as a diff. Used on two versions of the same diff, it
430 generates a diff that represents the diff from the first to the second
431 version.
432 </para>
434 <para>You can enable the <literal role="hg-ext">extdiff</literal> extension in the usual way, by
435 adding a line to the <literal role="rc-extensions">extensions</literal> section of your <filename role="special"> /.hgrc</filename>.
436 </para>
437 <programlisting>
438 <para> [extensions]
439 extdiff =
440 </para>
441 </programlisting>
442 <para>The <command>interdiff</command> command expects to be passed the names of two
443 files, but the <literal role="hg-ext">extdiff</literal> extension passes the program it runs a
444 pair of directories, each of which can contain an arbitrary number of
445 files. We thus need a small program that will run <command>interdiff</command>
446 on each pair of files in these two directories. This program is
447 available as <filename role="special">hg-interdiff</filename> in the <filename class="directory">examples</filename>
448 directory of the source code repository that accompanies this book.
449 <!-- &example.hg-interdiff; -->
450 </para>
452 <para>With the <filename role="special">hg-interdiff</filename> program in your shell's search path,
453 you can run it as follows, from inside an MQ patch directory:
454 </para>
455 <programlisting>
456 <para> hg extdiff -p hg-interdiff -r A:B my-change.patch
457 </para>
458 </programlisting>
459 <para>Since you'll probably want to use this long-winded command a lot, you
460 can get <literal role="hg-ext">hgext</literal> to make it available as a normal Mercurial
461 command, again by editing your <filename role="special"> /.hgrc</filename>.
462 </para>
463 <programlisting>
464 <para> [extdiff]
465 cmd.interdiff = hg-interdiff
466 </para>
467 </programlisting>
468 <para>This directs <literal role="hg-ext">hgext</literal> to make an <literal>interdiff</literal> command
469 available, so you can now shorten the previous invocation of
470 <command role="hg-ext-extdiff">extdiff</command> to something a little more wieldy.
471 </para>
472 <programlisting>
473 <para> hg interdiff -r A:B my-change.patch
474 </para>
475 </programlisting>
477 <note>
478 <para> The <command>interdiff</command> command works well only if the underlying
479 files against which versions of a patch are generated remain the
480 same. If you create a patch, modify the underlying files, and then
481 regenerate the patch, <command>interdiff</command> may not produce useful
482 output.
483 </para>
484 </note>
486 <para>The <literal role="hg-ext">extdiff</literal> extension is useful for more than merely improving
487 the presentation of MQ patches. To read more about it, go to
488 section <xref linkend="sec:hgext:extdiff"/>.
489 </para>
491 </sect2>
492 </sect1>
493 </chapter>
495 <!--
496 local variables:
497 sgml-parent-document: ("00book.xml" "book" "chapter")
498 end:
499 -->