| rev |
line source |
|
bos@108
|
1 \chapter{Behind the scenes}
|
|
jeffpc@56
|
2 \label{chap:concepts}
|
|
jeffpc@56
|
3
|
|
bos@108
|
4 Unlike many revision control systems, the concepts upon which
|
|
bos@108
|
5 Mercurial is built are simple enough that it's easy to understand how
|
|
bos@108
|
6 the software really works. Knowing this certainly isn't necessary,
|
|
bos@108
|
7 but I find it useful to have a ``mental model'' of what's going on.
|
|
jeffpc@56
|
8
|
|
bos@109
|
9 This understanding gives me confidence that Mercurial has been
|
|
bos@109
|
10 carefully designed to be both \emph{safe} and \emph{efficient}. And
|
|
bos@109
|
11 just as importantly, if I have a good idea what the software is doing
|
|
bos@109
|
12 when I perform a revision control task, I'm less likely to be
|
|
bos@109
|
13 surprised by its behaviour.
|
|
bos@109
|
14
|
|
bos@109
|
15 \section{Mercurial's historical record}
|
|
bos@109
|
16
|
|
bos@109
|
17 \subsection{Tracking the history of a single file}
|
|
jeffpc@56
|
18
|
|
bos@108
|
19 When Mercurial tracks modifications to a file, it stores the history
|
|
bos@108
|
20 of that file in a metadata object called a \emph{filelog}. Each entry
|
|
bos@108
|
21 in the filelog contains enough information to reconstruct one revision
|
|
bos@108
|
22 of the file that is being tracked. Filelogs are stored as files in
|
|
bos@108
|
23 the \sdirname{.hg/data} directory. A filelog contains two kinds of
|
|
bos@108
|
24 information: revision data, and an index to help Mercurial to find a
|
|
bos@108
|
25 revision efficiently.
|
|
jeffpc@56
|
26
|
|
bos@109
|
27 A file that is large, or has a lot of history, has its filelog stored
|
|
bos@109
|
28 in separate data (``\texttt{.d}'' suffix) and index (``\texttt{.i}''
|
|
bos@109
|
29 suffix) files. For small files without much history, the revision
|
|
bos@109
|
30 data and index are combined in a single ``\texttt{.i}'' file. The
|
|
bos@109
|
31 correspondence between a file in the working directory and the filelog
|
|
bos@109
|
32 that tracks its history in the repository is illustrated in
|
|
bos@109
|
33 figure~\ref{fig:concepts:filelog}.
|
|
jeffpc@56
|
34
|
|
jeffpc@56
|
35 \begin{figure}[ht]
|
|
bos@108
|
36 \centering
|
|
bos@108
|
37 \grafix{filelog}
|
|
bos@108
|
38 \caption{Relationships between files in working directory and
|
|
bos@108
|
39 filelogs in repository}
|
|
bos@108
|
40 \label{fig:concepts:filelog}
|
|
jeffpc@56
|
41 \end{figure}
|
|
jeffpc@56
|
42
|
|
bos@109
|
43 \subsection{Managing tracked files}
|
|
bos@109
|
44
|
|
bos@109
|
45 Mercurial uses a structure called a \emph{manifest} to collect
|
|
bos@109
|
46 together information about the files that it tracks. Each entry in
|
|
bos@109
|
47 the manifest contains information about the files present in a single
|
|
bos@109
|
48 changeset. An entry records which files are present in the changeset,
|
|
bos@109
|
49 the revision of each file, and a few other pieces of file metadata.
|
|
bos@109
|
50
|
|
bos@109
|
51 \subsection{Recording changeset information}
|
|
bos@109
|
52
|
|
bos@109
|
53 The \emph{changelog} contains information about each changeset. Each
|
|
bos@109
|
54 revision records who committed a change, the changeset comment, other
|
|
bos@109
|
55 pieces of changeset-related information, and the revision of the
|
|
bos@109
|
56 manifest to use.
|
|
bos@109
|
57
|
|
bos@109
|
58 \subsection{Relationships between revisions}
|
|
bos@109
|
59
|
|
bos@109
|
60 Within a changelog, a manifest, or a filelog, each revision stores a
|
|
bos@109
|
61 pointer to its immediate parent (or to its two parents, if it's a
|
|
bos@109
|
62 merge revision). As I mentioned above, there are also relationships
|
|
bos@109
|
63 between revisions \emph{across} these structures, and they are
|
|
bos@109
|
64 hierarchical in nature.
|
|
bos@109
|
65
|
|
bos@109
|
66 For every changeset in a repository, there is exactly one revision
|
|
bos@109
|
67 stored in the changelog. Each revision of the changelog contains a
|
|
bos@109
|
68 pointer to a single revision of the manifest. A revision of the
|
|
bos@109
|
69 manifest stores a pointer to a single revision of each filelog tracked
|
|
bos@109
|
70 when that changeset was created. These relationships are illustrated
|
|
bos@109
|
71 in figure~\ref{fig:concepts:metadata}.
|
|
bos@109
|
72
|
|
bos@109
|
73 \begin{figure}[ht]
|
|
bos@109
|
74 \centering
|
|
bos@109
|
75 \grafix{metadata}
|
|
bos@109
|
76 \caption{Metadata relationships}
|
|
bos@109
|
77 \label{fig:concepts:metadata}
|
|
bos@109
|
78 \end{figure}
|
|
bos@109
|
79
|
|
bos@109
|
80 Note that there is not a ``one to one'' relationship between revisions
|
|
bos@109
|
81 in these different metadata files. If the manifest hasn't changed
|
|
bos@109
|
82 between two changesets, their changelog entries will point to the same
|
|
bos@109
|
83 revision of the manifest. If a file that Mercurial tracks hasn't
|
|
bos@109
|
84 changed between two changesets, the entry for that file in the two
|
|
bos@109
|
85 revisions of the manifest will point to the same revision of its
|
|
bos@109
|
86 filelog.
|
|
bos@109
|
87
|
|
bos@109
|
88 \section{An efficient, unified, safe storage mechanism}
|
|
bos@109
|
89
|
|
bos@109
|
90 The underpinnings of changelogs, manifests, and filelogs are provided
|
|
bos@109
|
91 by a single structure called the \emph{revlog}.
|
|
bos@109
|
92
|
|
bos@109
|
93 \subsection{Efficient storage}
|
|
bos@109
|
94
|
|
bos@109
|
95 The revlog provides efficient storage of revisions using a
|
|
bos@109
|
96 \emph{delta} mechanism. Instead of storing a complete copy of a file
|
|
bos@109
|
97 for each revision, it stores the changes needed to transform an older
|
|
bos@109
|
98 revision into the new revision. For many kinds of file data, these
|
|
bos@109
|
99 deltas are typically a fraction of a percent of the size of a full
|
|
bos@109
|
100 copy of a file.
|
|
bos@109
|
101
|
|
bos@109
|
102 Some obsolete revision control systems can only work with deltas of
|
|
bos@109
|
103 text files. They must either store binary files as complete snapshots
|
|
bos@109
|
104 or encoded into a text representation, both of which are wasteful
|
|
bos@109
|
105 approaches. Mercurial can efficiently handle deltas of files with
|
|
bos@109
|
106 arbitrary binary contents; it doesn't need to treat text as special.
|
|
bos@109
|
107
|
|
bos@109
|
108 \subsection{Safe operation}
|
|
bos@109
|
109
|
|
bos@109
|
110 Mercurial only ever \emph{appends} data to the end of a revlog file.
|
|
bos@109
|
111 It never modifies a section of a file after it has written it. This
|
|
bos@109
|
112 is both more robust and efficient than schemes that need to modify or
|
|
bos@109
|
113 rewrite data.
|
|
bos@109
|
114
|
|
bos@109
|
115 In addition, Mercurial treats every write as part of a
|
|
bos@109
|
116 \emph{transaction} that can span a number of files. A transaction is
|
|
bos@109
|
117 \emph{atomic}: either the entire transaction succeeds and its effects
|
|
bos@109
|
118 are all visible to readers in one go, or the whole thing is undone.
|
|
bos@109
|
119 This guarantee of atomicity means that if you're running two copies of
|
|
bos@109
|
120 Mercurial, where one is reading data and one is writing it, the reader
|
|
bos@109
|
121 will never see a partially written result that might confuse it.
|
|
bos@109
|
122
|
|
bos@109
|
123 The fact that Mercurial only appends to files makes it easier to
|
|
bos@109
|
124 provide this transactional guarantee. The easier it is to do stuff
|
|
bos@109
|
125 like this, the more confident you should be that it's done correctly.
|
|
bos@109
|
126
|
|
bos@109
|
127 \subsection{Fast retrieval}
|
|
bos@109
|
128
|
|
bos@109
|
129 Mercurial cleverly avoids a pitfall common to all earlier
|
|
bos@109
|
130 revision control systems: the problem of \emph{inefficient retrieval}.
|
|
bos@109
|
131 Most revision control systems store the contents of a revision as an
|
|
bos@109
|
132 incremental series of modifications against a ``snapshot''. To
|
|
bos@109
|
133 reconstruct a specific revision, you must first read the snapshot, and
|
|
bos@109
|
134 then every one of the revisions between the snapshot and your target
|
|
bos@109
|
135 revision. The more history that a file accumulates, the more
|
|
bos@109
|
136 revisions you must read, hence the longer it takes to reconstruct a
|
|
bos@109
|
137 particular revision.
|
|
bos@109
|
138
|
|
bos@109
|
139 The innovation that Mercurial applies to this problem is simple but
|
|
bos@109
|
140 effective. Once the cumulative amount of delta information stored
|
|
bos@109
|
141 since the last snapshot exceeds a fixed threshold, it stores a new
|
|
bos@109
|
142 snapshot (compressed, of course), instead of another delta. This
|
|
bos@109
|
143 makes it possible to reconstruct \emph{any} revision of a file
|
|
bos@109
|
144 quickly. This approach works so well that it has subsequently been
|
|
bos@109
|
145 copied by several other revision control systems.
|
|
bos@109
|
146
|
|
bos@109
|
147 \subsubsection{Aside: the influence of video compression}
|
|
bos@109
|
148
|
|
bos@109
|
149 If you're familiar with video compression or have ever watched a TV
|
|
bos@109
|
150 feed through a digital cable or satellite service, you may know that
|
|
bos@109
|
151 most video compression schemes store each frame of video as a delta
|
|
bos@109
|
152 against its predecessor frame. In addition, these schemes use
|
|
bos@109
|
153 ``lossy'' compression techniques to increase the compression ratio, so
|
|
bos@109
|
154 visual errors accumulate over the course of a number of inter-frame
|
|
bos@109
|
155 deltas.
|
|
bos@109
|
156
|
|
bos@109
|
157 Because it's possible for a video stream to ``drop out'' occasionally
|
|
bos@109
|
158 due to signal glitches, and to limit the accumulation of artefacts
|
|
bos@109
|
159 introduced by the lossy compression process, video encoders
|
|
bos@109
|
160 periodically insert a complete frame (called a ``key frame'') into the
|
|
bos@109
|
161 video stream; the next delta is generated against that frame. This
|
|
bos@109
|
162 means that if the video signal gets interrupted, it will resume once
|
|
bos@109
|
163 the next key frame is received. Also, the accumulation of encoding
|
|
bos@109
|
164 errors restarts anew with each key frame.
|
|
bos@109
|
165
|
|
bos@109
|
166 \subsection{Clever compression}
|
|
bos@109
|
167
|
|
bos@109
|
168 When appropriate, Mercurial will store both snapshots and deltas in
|
|
bos@109
|
169 compressed form. It does this by always \emph{trying to} compress a
|
|
bos@109
|
170 snapshot or delta, but only storing the compressed version if it's
|
|
bos@109
|
171 smaller than the uncompressed version.
|
|
bos@109
|
172
|
|
bos@109
|
173 This means that Mercurial does ``the right thing'' when storing a file
|
|
bos@109
|
174 whose native form is compressed, such as a \texttt{zip} archive or a
|
|
bos@109
|
175 JPEG image. When these types of files are compressed a second time,
|
|
bos@109
|
176 the resulting file is usually bigger than the once-compressed form,
|
|
bos@109
|
177 and so Mercurial will store the plain \texttt{zip} or JPEG.
|
|
bos@109
|
178
|
|
bos@109
|
179 Deltas between revisions of a compressed file are usually larger than
|
|
bos@109
|
180 snapshots of the file, and Mercurial again does ``the right thing'' in
|
|
bos@109
|
181 these cases. It finds that such a delta exceeds the threshold at
|
|
bos@109
|
182 which it should store a complete snapshot of the file, so it stores
|
|
bos@109
|
183 the snapshot, again saving space compared to a naive delta-only
|
|
bos@109
|
184 approach.
|
|
bos@109
|
185
|
|
bos@109
|
186 \subsection{Strong integrity}
|
|
bos@109
|
187
|
|
bos@109
|
188 Along with delta or snapshot information, a revlog entry contains a
|
|
bos@109
|
189 cryptographic hash of the data that it represents. This makes it
|
|
bos@109
|
190 difficult to forge the contents of a revision, and easy to detect
|
|
bos@109
|
191 accidental corruption.
|
|
bos@109
|
192
|
|
bos@109
|
193 Mercurial checks these hashes when retrieving file revisions and when
|
|
bos@109
|
194 pulling changes from a repository. If it encounters an integrity
|
|
bos@109
|
195 problem, it will complain and stop whatever it's doing.
|
|
bos@109
|
196
|
|
bos@109
|
197 In addition to the effect it has on retrieval efficiency, Mercurial's
|
|
bos@109
|
198 use of periodic snapshots makes it more robust against partial data
|
|
bos@109
|
199 corruption. If a revlog becomes partly corrupted due to a hardware
|
|
bos@109
|
200 error or system bug, it's often possible to reconstruct some or most
|
|
bos@109
|
201 revisions from the uncorrupted sections of the revlog, both before and
|
|
bos@109
|
202 after the corrupted section. This would not be possible with a
|
|
bos@109
|
203 delta-only storage model.
|
|
bos@109
|
204
|
|
bos@109
|
205 \subsection{Read/write ordering and atomicity}
|
|
bos@109
|
206
|
|
bos@109
|
207 Appending to files isn't the whole story when it comes to guaranteeing
|
|
bos@109
|
208 that a reader won't see a partial write. If you recall
|
|
bos@109
|
209 figure~\ref{fig:concepts:metadata}, revisions in the changelog point to
|
|
bos@109
|
210 revisions in the manifest, and revisions in the manifest point to
|
|
bos@109
|
211 revisions in filelogs. This hierarchy is deliberate.
|
|
bos@109
|
212
|
|
bos@109
|
213 A writer starts a transaction by writing filelog and manifest data,
|
|
bos@109
|
214 and doesn't write any changelog data until those are finished. A
|
|
bos@109
|
215 reader starts by reading changelog data, then manifest data, followed
|
|
bos@109
|
216 by filelog data.
|
|
bos@109
|
217
|
|
bos@109
|
218 Since the writer has always finished writing filelog and manifest data
|
|
bos@109
|
219 before it writes to the changelog, a reader will never read a pointer
|
|
bos@109
|
220 to a partially written manifest revision from the changelog, and it will
|
|
bos@109
|
221 never read a pointer to a partially written filelog revision from the
|
|
bos@109
|
222 manifest.
|
|
bos@109
|
223
|
|
bos@109
|
224 \subsection{Concurrent access}
|
|
bos@109
|
225
|
|
bos@109
|
226 The read/write ordering and atomicity guarantees mean that Mercurial
|
|
bos@109
|
227 never needs to \emph{lock} a repository when it's reading data, even
|
|
bos@109
|
228 if the repository is being written to while the read is occurring.
|
|
bos@109
|
229 This has a big effect on scalability; you can have an arbitrary number
|
|
bos@109
|
230 of Mercurial processes safely reading data from a repository safely
|
|
bos@109
|
231 all at once, no matter whether it's being written to or not.
|
|
bos@109
|
232
|
|
bos@109
|
233 The lockless nature of reading means that if you're sharing a
|
|
bos@109
|
234 repository on a multi-user system, you don't need to grant other local
|
|
bos@109
|
235 users permission to \emph{write} to your repository in order for them
|
|
bos@109
|
236 to be able to clone it or pull changes from it; they only need
|
|
bos@109
|
237 \emph{read} permission. (This is \emph{not} a common feature among
|
|
bos@109
|
238 revision control systems, so don't take it for granted! Most require
|
|
bos@109
|
239 readers to be able to lock a repository to access it safely, and this
|
|
bos@109
|
240 requires write permission on at least one directory, which of course
|
|
bos@109
|
241 makes for all kinds of nasty and annoying security and administrative
|
|
bos@109
|
242 problems.)
|
|
bos@109
|
243
|
|
bos@109
|
244 Mercurial uses a locking mechanism to ensure that only one process can
|
|
bos@109
|
245 write to a repository at a time. This locking mechanism is safe even
|
|
bos@109
|
246 over filesystems that are notoriously unsafe for locking, such as NFS.
|
|
bos@109
|
247 If a repository is locked, a writer will wait for a while to retry if
|
|
bos@109
|
248 the repository becomes unlocked, but if the repository remains locked
|
|
bos@109
|
249 for too long, the process attempting to write will time out after a
|
|
bos@109
|
250 while. This means that your daily automated scripts won't get stuck
|
|
bos@109
|
251 forever and pile up if a system crashes unnoticed, for example. (Yes,
|
|
bos@109
|
252 the timeout is configurable, from zero to infinity.)
|
|
bos@109
|
253
|
|
bos@109
|
254
|
|
bos@109
|
255
|
|
jeffpc@56
|
256 %%% Local Variables:
|
|
jeffpc@56
|
257 %%% mode: latex
|
|
jeffpc@56
|
258 %%% TeX-master: "00book"
|
|
jeffpc@56
|
259 %%% End:
|
