hgbook
view en/ch13-mq-collab.xml @ 567:8fcd44708f41
Uncomment all the mangled interaction examples.
| author | Bryan O'Sullivan <bos@serpentine.com> |
|---|---|
| date | Mon Mar 09 23:22:09 2009 -0700 (2009-03-09) |
| parents | b90b024729f1 |
| children | 13513d2a128d |
line source
1 <!-- vim: set filetype=docbkxml shiftwidth=2 autoindent expandtab tw=77 : -->
3 <chapter id="chap:mq-collab">
4 <title>Advanced uses of Mercurial Queues</title>
6 <para>While it's easy to pick up straightforward uses of Mercurial
7 Queues, use of a little discipline and some of MQ's less
8 frequently used capabilities makes it possible to work in
9 complicated development environments.</para>
11 <para>In this chapter, I will use as an example a technique I have
12 used to manage the development of an Infiniband device driver for
13 the Linux kernel. The driver in question is large (at least as
14 drivers go), with 25,000 lines of code spread across 35 source
15 files. It is maintained by a small team of developers.</para>
17 <para>While much of the material in this chapter is specific to
18 Linux, the same principles apply to any code base for which you're
19 not the primary owner, and upon which you need to do a lot of
20 development.</para>
22 <sect1>
23 <title>The problem of many targets</title>
25 <para>The Linux kernel changes rapidly, and has never been
26 internally stable; developers frequently make drastic changes
27 between releases. This means that a version of the driver that
28 works well with a particular released version of the kernel will
29 not even <emphasis>compile</emphasis> correctly against,
30 typically, any other version.</para>
32 <para>To maintain a driver, we have to keep a number of distinct
33 versions of Linux in mind.</para>
34 <itemizedlist>
35 <listitem><para>One target is the main Linux kernel development
36 tree. Maintenance of the code is in this case partly shared
37 by other developers in the kernel community, who make
38 <quote>drive-by</quote> modifications to the driver as they
39 develop and refine kernel subsystems.</para>
40 </listitem>
41 <listitem><para>We also maintain a number of
42 <quote>backports</quote> to older versions of the Linux
43 kernel, to support the needs of customers who are running
44 older Linux distributions that do not incorporate our
45 drivers. (To <emphasis>backport</emphasis> a piece of code
46 is to modify it to work in an older version of its target
47 environment than the version it was developed for.)</para>
48 </listitem>
49 <listitem><para>Finally, we make software releases on a schedule
50 that is necessarily not aligned with those used by Linux
51 distributors and kernel developers, so that we can deliver
52 new features to customers without forcing them to upgrade
53 their entire kernels or distributions.</para>
54 </listitem></itemizedlist>
56 <sect2>
57 <title>Tempting approaches that don't work well</title>
59 <para>There are two <quote>standard</quote> ways to maintain a
60 piece of software that has to target many different
61 environments.</para>
63 <para>The first is to maintain a number of branches, each
64 intended for a single target. The trouble with this approach
65 is that you must maintain iron discipline in the flow of
66 changes between repositories. A new feature or bug fix must
67 start life in a <quote>pristine</quote> repository, then
68 percolate out to every backport repository. Backport changes
69 are more limited in the branches they should propagate to; a
70 backport change that is applied to a branch where it doesn't
71 belong will probably stop the driver from compiling.</para>
73 <para>The second is to maintain a single source tree filled with
74 conditional statements that turn chunks of code on or off
75 depending on the intended target. Because these
76 <quote>ifdefs</quote> are not allowed in the Linux kernel
77 tree, a manual or automatic process must be followed to strip
78 them out and yield a clean tree. A code base maintained in
79 this fashion rapidly becomes a rat's nest of conditional
80 blocks that are difficult to understand and maintain.</para>
82 <para>Neither of these approaches is well suited to a situation
83 where you don't <quote>own</quote> the canonical copy of a
84 source tree. In the case of a Linux driver that is
85 distributed with the standard kernel, Linus's tree contains
86 the copy of the code that will be treated by the world as
87 canonical. The upstream version of <quote>my</quote> driver
88 can be modified by people I don't know, without me even
89 finding out about it until after the changes show up in
90 Linus's tree.</para>
92 <para>These approaches have the added weakness of making it
93 difficult to generate well-formed patches to submit
94 upstream.</para>
96 <para>In principle, Mercurial Queues seems like a good candidate
97 to manage a development scenario such as the above. While
98 this is indeed the case, MQ contains a few added features that
99 make the job more pleasant.</para>
101 </sect2>
102 </sect1>
103 <sect1>
104 <title>Conditionally applying patches with guards</title>
106 <para>Perhaps the best way to maintain sanity with so many targets
107 is to be able to choose specific patches to apply for a given
108 situation. MQ provides a feature called <quote>guards</quote>
109 (which originates with quilt's <literal>guards</literal>
110 command) that does just this. To start off, let's create a
111 simple repository for experimenting in.</para>
113 &interaction.mq.guards.init;
115 <para>This gives us a tiny repository that contains two patches
116 that don't have any dependencies on each other, because they
117 touch different files.</para>
119 <para>The idea behind conditional application is that you can
120 <quote>tag</quote> a patch with a <emphasis>guard</emphasis>,
121 which is simply a text string of your choosing, then tell MQ to
122 select specific guards to use when applying patches. MQ will
123 then either apply, or skip over, a guarded patch, depending on
124 the guards that you have selected.</para>
126 <para>A patch can have an arbitrary number of guards; each one is
127 <emphasis>positive</emphasis> (<quote>apply this patch if this
128 guard is selected</quote>) or <emphasis>negative</emphasis>
129 (<quote>skip this patch if this guard is selected</quote>). A
130 patch with no guards is always applied.</para>
132 </sect1>
133 <sect1>
134 <title>Controlling the guards on a patch</title>
136 <para>The <command role="hg-ext-mq">qguard</command> command lets
137 you determine which guards should apply to a patch, or display
138 the guards that are already in effect. Without any arguments, it
139 displays the guards on the current topmost patch.</para>
141 &interaction.mq.guards.qguard;
143 <para>To set a positive guard on a patch, prefix the name of the
144 guard with a <quote><literal>+</literal></quote>.</para>
146 &interaction.mq.guards.qguard.pos;
148 <para>To set a negative guard
149 on a patch, prefix the name of the guard with a
150 <quote><literal>-</literal></quote>.</para>
152 &interaction.mq.guards.qguard.neg;
154 <note>
155 <para> The <command role="hg-ext-mq">qguard</command> command
156 <emphasis>sets</emphasis> the guards on a patch; it doesn't
157 <emphasis>modify</emphasis> them. What this means is that if
158 you run <command role="hg-cmd">hg qguard +a +b</command> on a
159 patch, then <command role="hg-cmd">hg qguard +c</command> on
160 the same patch, the <emphasis>only</emphasis> guard that will
161 be set on it afterwards is <literal>+c</literal>.</para>
162 </note>
164 <para>Mercurial stores guards in the <filename
165 role="special">series</filename> file; the form in which they
166 are stored is easy both to understand and to edit by hand. (In
167 other words, you don't have to use the <command
168 role="hg-ext-mq">qguard</command> command if you don't want
169 to; it's okay to simply edit the <filename
170 role="special">series</filename> file.)</para>
172 &interaction.mq.guards.series;
174 </sect1>
175 <sect1>
176 <title>Selecting the guards to use</title>
178 <para>The <command role="hg-ext-mq">qselect</command> command
179 determines which guards are active at a given time. The effect
180 of this is to determine which patches MQ will apply the next
181 time you run <command role="hg-ext-mq">qpush</command>. It has
182 no other effect; in particular, it doesn't do anything to
183 patches that are already applied.</para>
185 <para>With no arguments, the <command
186 role="hg-ext-mq">qselect</command> command lists the guards
187 currently in effect, one per line of output. Each argument is
188 treated as the name of a guard to apply.</para>
190 &interaction.mq.guards.qselect.foo;
192 <para>In case you're interested, the currently selected guards are
193 stored in the <filename role="special">guards</filename> file.</para>
195 &interaction.mq.guards.qselect.cat;
197 <para>We can see the effect the selected guards have when we run
198 <command role="hg-ext-mq">qpush</command>.</para>
200 &interaction.mq.guards.qselect.qpush;
202 <para>A guard cannot start with a
203 <quote><literal>+</literal></quote> or
204 <quote><literal>-</literal></quote> character. The name of a
205 guard must not contain white space, but most other characters
206 are acceptable. If you try to use a guard with an invalid name,
207 MQ will complain:</para>
209 &interaction.mq.guards.qselect.error;
211 <para>Changing the selected guards changes the patches that are
212 applied.</para>
214 &interaction.mq.guards.qselect.quux;
216 <para>You can see in the example below that negative guards take
217 precedence over positive guards.</para>
219 &interaction.mq.guards.qselect.foobar;
221 </sect1>
222 <sect1>
223 <title>MQ's rules for applying patches</title>
225 <para>The rules that MQ uses when deciding whether to apply a
226 patch are as follows.</para>
227 <itemizedlist>
228 <listitem><para>A patch that has no guards is always
229 applied.</para>
230 </listitem>
231 <listitem><para>If the patch has any negative guard that matches
232 any currently selected guard, the patch is skipped.</para>
233 </listitem>
234 <listitem><para>If the patch has any positive guard that matches
235 any currently selected guard, the patch is applied.</para>
236 </listitem>
237 <listitem><para>If the patch has positive or negative guards,
238 but none matches any currently selected guard, the patch is
239 skipped.</para>
240 </listitem></itemizedlist>
242 </sect1>
243 <sect1>
244 <title>Trimming the work environment</title>
246 <para>In working on the device driver I mentioned earlier, I don't
247 apply the patches to a normal Linux kernel tree. Instead, I use
248 a repository that contains only a snapshot of the source files
249 and headers that are relevant to Infiniband development. This
250 repository is 1% the size of a kernel repository, so it's easier
251 to work with.</para>
253 <para>I then choose a <quote>base</quote> version on top of which
254 the patches are applied. This is a snapshot of the Linux kernel
255 tree as of a revision of my choosing. When I take the snapshot,
256 I record the changeset ID from the kernel repository in the
257 commit message. Since the snapshot preserves the
258 <quote>shape</quote> and content of the relevant parts of the
259 kernel tree, I can apply my patches on top of either my tiny
260 repository or a normal kernel tree.</para>
262 <para>Normally, the base tree atop which the patches apply should
263 be a snapshot of a very recent upstream tree. This best
264 facilitates the development of patches that can easily be
265 submitted upstream with few or no modifications.</para>
267 </sect1>
268 <sect1>
269 <title>Dividing up the <filename role="special">series</filename>
270 file</title>
272 <para>I categorise the patches in the <filename
273 role="special">series</filename> file into a number of logical
274 groups. Each section of like patches begins with a block of
275 comments that describes the purpose of the patches that
276 follow.</para>
278 <para>The sequence of patch groups that I maintain follows. The
279 ordering of these groups is important; I'll describe why after I
280 introduce the groups.</para>
281 <itemizedlist>
282 <listitem><para>The <quote>accepted</quote> group. Patches that
283 the development team has submitted to the maintainer of the
284 Infiniband subsystem, and which he has accepted, but which
285 are not present in the snapshot that the tiny repository is
286 based on. These are <quote>read only</quote> patches,
287 present only to transform the tree into a similar state as
288 it is in the upstream maintainer's repository.</para>
289 </listitem>
290 <listitem><para>The <quote>rework</quote> group. Patches that I
291 have submitted, but that the upstream maintainer has
292 requested modifications to before he will accept
293 them.</para>
294 </listitem>
295 <listitem><para>The <quote>pending</quote> group. Patches that
296 I have not yet submitted to the upstream maintainer, but
297 which we have finished working on. These will be <quote>read
298 only</quote> for a while. If the upstream maintainer
299 accepts them upon submission, I'll move them to the end of
300 the <quote>accepted</quote> group. If he requests that I
301 modify any, I'll move them to the beginning of the
302 <quote>rework</quote> group.</para>
303 </listitem>
304 <listitem><para>The <quote>in progress</quote> group. Patches
305 that are actively being developed, and should not be
306 submitted anywhere yet.</para>
307 </listitem>
308 <listitem><para>The <quote>backport</quote> group. Patches that
309 adapt the source tree to older versions of the kernel
310 tree.</para>
311 </listitem>
312 <listitem><para>The <quote>do not ship</quote> group. Patches
313 that for some reason should never be submitted upstream.
314 For example, one such patch might change embedded driver
315 identification strings to make it easier to distinguish, in
316 the field, between an out-of-tree version of the driver and
317 a version shipped by a distribution vendor.</para>
318 </listitem></itemizedlist>
320 <para>Now to return to the reasons for ordering groups of patches
321 in this way. We would like the lowest patches in the stack to
322 be as stable as possible, so that we will not need to rework
323 higher patches due to changes in context. Putting patches that
324 will never be changed first in the <filename
325 role="special">series</filename> file serves this
326 purpose.</para>
328 <para>We would also like the patches that we know we'll need to
329 modify to be applied on top of a source tree that resembles the
330 upstream tree as closely as possible. This is why we keep
331 accepted patches around for a while.</para>
333 <para>The <quote>backport</quote> and <quote>do not ship</quote>
334 patches float at the end of the <filename
335 role="special">series</filename> file. The backport patches
336 must be applied on top of all other patches, and the <quote>do
337 not ship</quote> patches might as well stay out of harm's
338 way.</para>
340 </sect1>
341 <sect1>
342 <title>Maintaining the patch series</title>
344 <para>In my work, I use a number of guards to control which
345 patches are to be applied.</para>
347 <itemizedlist>
348 <listitem><para><quote>Accepted</quote> patches are guarded with
349 <literal>accepted</literal>. I enable this guard most of
350 the time. When I'm applying the patches on top of a tree
351 where the patches are already present, I can turn this patch
352 off, and the patches that follow it will apply
353 cleanly.</para>
354 </listitem>
355 <listitem><para>Patches that are <quote>finished</quote>, but
356 not yet submitted, have no guards. If I'm applying the
357 patch stack to a copy of the upstream tree, I don't need to
358 enable any guards in order to get a reasonably safe source
359 tree.</para>
360 </listitem>
361 <listitem><para>Those patches that need reworking before being
362 resubmitted are guarded with
363 <literal>rework</literal>.</para>
364 </listitem>
365 <listitem><para>For those patches that are still under
366 development, I use <literal>devel</literal>.</para>
367 </listitem>
368 <listitem><para>A backport patch may have several guards, one
369 for each version of the kernel to which it applies. For
370 example, a patch that backports a piece of code to 2.6.9
371 will have a <literal>2.6.9</literal> guard.</para>
372 </listitem></itemizedlist>
373 <para>This variety of guards gives me considerable flexibility in
374 determining what kind of source tree I want to end up with. For
375 most situations, the selection of appropriate guards is
376 automated during the build process, but I can manually tune the
377 guards to use for less common circumstances.</para>
379 <sect2>
380 <title>The art of writing backport patches</title>
382 <para>Using MQ, writing a backport patch is a simple process.
383 All such a patch has to do is modify a piece of code that uses
384 a kernel feature not present in the older version of the
385 kernel, so that the driver continues to work correctly under
386 that older version.</para>
388 <para>A useful goal when writing a good backport patch is to
389 make your code look as if it was written for the older version
390 of the kernel you're targeting. The less obtrusive the patch,
391 the easier it will be to understand and maintain. If you're
392 writing a collection of backport patches to avoid the
393 <quote>rat's nest</quote> effect of lots of
394 <literal>#ifdef</literal>s (hunks of source code that are only
395 used conditionally) in your code, don't introduce
396 version-dependent <literal>#ifdef</literal>s into the patches.
397 Instead, write several patches, each of which makes
398 unconditional changes, and control their application using
399 guards.</para>
401 <para>There are two reasons to divide backport patches into a
402 distinct group, away from the <quote>regular</quote> patches
403 whose effects they modify. The first is that intermingling the
404 two makes it more difficult to use a tool like the <literal
405 role="hg-ext">patchbomb</literal> extension to automate the
406 process of submitting the patches to an upstream maintainer.
407 The second is that a backport patch could perturb the context
408 in which a subsequent regular patch is applied, making it
409 impossible to apply the regular patch cleanly
410 <emphasis>without</emphasis> the earlier backport patch
411 already being applied.</para>
413 </sect2>
414 </sect1>
415 <sect1>
416 <title>Useful tips for developing with MQ</title>
418 <sect2>
419 <title>Organising patches in directories</title>
421 <para>If you're working on a substantial project with MQ, it's
422 not difficult to accumulate a large number of patches. For
423 example, I have one patch repository that contains over 250
424 patches.</para>
426 <para>If you can group these patches into separate logical
427 categories, you can if you like store them in different
428 directories; MQ has no problems with patch names that contain
429 path separators.</para>
431 </sect2>
432 <sect2 id="mq-collab:tips:interdiff">
433 <title>Viewing the history of a patch</title>
435 <para>If you're developing a set of patches over a long time,
436 it's a good idea to maintain them in a repository, as
437 discussed in section <xref linkend="sec:mq:repo"/>. If you do
438 so, you'll quickly
439 discover that using the <command role="hg-cmd">hg
440 diff</command> command to look at the history of changes to
441 a patch is unworkable. This is in part because you're looking
442 at the second derivative of the real code (a diff of a diff),
443 but also because MQ adds noise to the process by modifying
444 time stamps and directory names when it updates a
445 patch.</para>
447 <para>However, you can use the <literal
448 role="hg-ext">extdiff</literal> extension, which is bundled
449 with Mercurial, to turn a diff of two versions of a patch into
450 something readable. To do this, you will need a third-party
451 package called <literal role="package">patchutils</literal>
452 <citation>web:patchutils</citation>. This provides a command
453 named <command>interdiff</command>, which shows the
454 differences between two diffs as a diff. Used on two versions
455 of the same diff, it generates a diff that represents the diff
456 from the first to the second version.</para>
458 <para>You can enable the <literal
459 role="hg-ext">extdiff</literal> extension in the usual way,
460 by adding a line to the <literal
461 role="rc-extensions">extensions</literal> section of your
462 <filename role="special"> /.hgrc</filename>.</para>
463 <programlisting>[extensions] extdiff =</programlisting>
464 <para>The <command>interdiff</command> command expects to be
465 passed the names of two files, but the <literal
466 role="hg-ext">extdiff</literal> extension passes the program
467 it runs a pair of directories, each of which can contain an
468 arbitrary number of files. We thus need a small program that
469 will run <command>interdiff</command> on each pair of files in
470 these two directories. This program is available as <filename
471 role="special">hg-interdiff</filename> in the <filename
472 class="directory">examples</filename> directory of the
473 source code repository that accompanies this book. <!--
474 &example.hg-interdiff; --></para>
476 <para>With the <filename role="special">hg-interdiff</filename>
477 program in your shell's search path, you can run it as
478 follows, from inside an MQ patch directory:</para>
479 <programlisting>hg extdiff -p hg-interdiff -r A:B
480 my-change.patch</programlisting>
481 <para>Since you'll probably want to use this long-winded command
482 a lot, you can get <literal role="hg-ext">hgext</literal> to
483 make it available as a normal Mercurial command, again by
484 editing your <filename role="special">
485 /.hgrc</filename>.</para>
486 <programlisting>[extdiff] cmd.interdiff =
487 hg-interdiff</programlisting>
488 <para>This directs <literal role="hg-ext">hgext</literal> to
489 make an <literal>interdiff</literal> command available, so you
490 can now shorten the previous invocation of <command
491 role="hg-ext-extdiff">extdiff</command> to something a
492 little more wieldy.</para>
493 <programlisting>hg interdiff -r A:B
494 my-change.patch</programlisting>
496 <note>
497 <para> The <command>interdiff</command> command works well
498 only if the underlying files against which versions of a
499 patch are generated remain the same. If you create a patch,
500 modify the underlying files, and then regenerate the patch,
501 <command>interdiff</command> may not produce useful
502 output.</para>
503 </note>
505 <para>The <literal role="hg-ext">extdiff</literal> extension is
506 useful for more than merely improving the presentation of MQ
507 patches. To read more about it, go to section <xref
508 linkend="sec:hgext:extdiff"/>.</para>
510 </sect2>
511 </sect1>
512 </chapter>
514 <!--
515 local variables:
516 sgml-parent-document: ("00book.xml" "book" "chapter")
517 end:
518 -->