bos@108: \chapter{Behind the scenes}
jeffpc@56: \label{chap:concepts}
jeffpc@56: 
bos@108: Unlike many revision control systems, the concepts upon which
bos@108: Mercurial is built are simple enough that it's easy to understand how
bos@108: the software really works.  Knowing this certainly isn't necessary,
bos@108: but I find it useful to have a ``mental model'' of what's going on.
jeffpc@56: 
bos@109: This understanding gives me confidence that Mercurial has been
bos@109: carefully designed to be both \emph{safe} and \emph{efficient}.  And
bos@109: just as importantly, if I have a good idea what the software is doing
bos@109: when I perform a revision control task, I'm less likely to be
bos@109: surprised by its behaviour.
bos@109: 
bos@109: \section{Mercurial's historical record}
bos@109: 
bos@109: \subsection{Tracking the history of a single file}
jeffpc@56: 
bos@108: When Mercurial tracks modifications to a file, it stores the history
bos@108: of that file in a metadata object called a \emph{filelog}.  Each entry
bos@108: in the filelog contains enough information to reconstruct one revision
bos@108: of the file that is being tracked.  Filelogs are stored as files in
bos@108: the \sdirname{.hg/data} directory.  A filelog contains two kinds of
bos@108: information: revision data, and an index to help Mercurial to find a
bos@108: revision efficiently.  
jeffpc@56: 
bos@109: A file that is large, or has a lot of history, has its filelog stored
bos@109: in separate data (``\texttt{.d}'' suffix) and index (``\texttt{.i}''
bos@109: suffix) files.  For small files without much history, the revision
bos@109: data and index are combined in a single ``\texttt{.i}'' file.  The
bos@109: correspondence between a file in the working directory and the filelog
bos@109: that tracks its history in the repository is illustrated in
bos@109: figure~\ref{fig:concepts:filelog}.
jeffpc@56: 
jeffpc@56: \begin{figure}[ht]
bos@108:   \centering
bos@108:   \grafix{filelog}
bos@108:   \caption{Relationships between files in working directory and
bos@108:     filelogs in repository}
bos@108:   \label{fig:concepts:filelog}
jeffpc@56: \end{figure}
jeffpc@56: 
bos@109: \subsection{Managing tracked files}
bos@109: 
bos@109: Mercurial uses a structure called a \emph{manifest} to collect
bos@109: together information about the files that it tracks.  Each entry in
bos@109: the manifest contains information about the files present in a single
bos@109: changeset.  An entry records which files are present in the changeset,
bos@109: the revision of each file, and a few other pieces of file metadata.
bos@109: 
bos@109: \subsection{Recording changeset information}
bos@109: 
bos@109: The \emph{changelog} contains information about each changeset.  Each
bos@109: revision records who committed a change, the changeset comment, other
bos@109: pieces of changeset-related information, and the revision of the
bos@109: manifest to use.
bos@109: 
bos@109: \subsection{Relationships between revisions}
bos@109: 
bos@109: Within a changelog, a manifest, or a filelog, each revision stores a
bos@109: pointer to its immediate parent (or to its two parents, if it's a
bos@109: merge revision).  As I mentioned above, there are also relationships
bos@109: between revisions \emph{across} these structures, and they are
bos@109: hierarchical in nature.
bos@109: 
bos@109: For every changeset in a repository, there is exactly one revision
bos@109: stored in the changelog.  Each revision of the changelog contains a
bos@109: pointer to a single revision of the manifest.  A revision of the
bos@109: manifest stores a pointer to a single revision of each filelog tracked
bos@109: when that changeset was created.  These relationships are illustrated
bos@109: in figure~\ref{fig:concepts:metadata}.
bos@109: 
bos@109: \begin{figure}[ht]
bos@109:   \centering
bos@109:   \grafix{metadata}
bos@109:   \caption{Metadata relationships}
bos@109:   \label{fig:concepts:metadata}
bos@109: \end{figure}
bos@109: 
bos@109: Note that there is not a ``one to one'' relationship between revisions
bos@109: in these different metadata files.  If the manifest hasn't changed
bos@109: between two changesets, their changelog entries will point to the same
bos@109: revision of the manifest.  If a file that Mercurial tracks hasn't
bos@109: changed between two changesets, the entry for that file in the two
bos@109: revisions of the manifest will point to the same revision of its
bos@109: filelog.
bos@109: 
bos@109: \section{An efficient, unified, safe storage mechanism}
bos@109: 
bos@109: The underpinnings of changelogs, manifests, and filelogs are provided
bos@109: by a single structure called the \emph{revlog}.
bos@109: 
bos@109: \subsection{Efficient storage}
bos@109: 
bos@109: The revlog provides efficient storage of revisions using a
bos@109: \emph{delta} mechanism.  Instead of storing a complete copy of a file
bos@109: for each revision, it stores the changes needed to transform an older
bos@109: revision into the new revision.  For many kinds of file data, these
bos@109: deltas are typically a fraction of a percent of the size of a full
bos@109: copy of a file.
bos@109: 
bos@109: Some obsolete revision control systems can only work with deltas of
bos@109: text files.  They must either store binary files as complete snapshots
bos@109: or encoded into a text representation, both of which are wasteful
bos@109: approaches.  Mercurial can efficiently handle deltas of files with
bos@109: arbitrary binary contents; it doesn't need to treat text as special.
bos@109: 
bos@109: \subsection{Safe operation}
bos@109: 
bos@109: Mercurial only ever \emph{appends} data to the end of a revlog file.
bos@109: It never modifies a section of a file after it has written it.  This
bos@109: is both more robust and efficient than schemes that need to modify or
bos@109: rewrite data.
bos@109: 
bos@109: In addition, Mercurial treats every write as part of a
bos@109: \emph{transaction} that can span a number of files.  A transaction is
bos@109: \emph{atomic}: either the entire transaction succeeds and its effects
bos@109: are all visible to readers in one go, or the whole thing is undone.
bos@109: This guarantee of atomicity means that if you're running two copies of
bos@109: Mercurial, where one is reading data and one is writing it, the reader
bos@109: will never see a partially written result that might confuse it.
bos@109: 
bos@109: The fact that Mercurial only appends to files makes it easier to
bos@109: provide this transactional guarantee.  The easier it is to do stuff
bos@109: like this, the more confident you should be that it's done correctly.
bos@109: 
bos@109: \subsection{Fast retrieval}
bos@109: 
bos@109: Mercurial cleverly avoids a pitfall common to all earlier
bos@109: revision control systems: the problem of \emph{inefficient retrieval}.
bos@109: Most revision control systems store the contents of a revision as an
bos@109: incremental series of modifications against a ``snapshot''.  To
bos@109: reconstruct a specific revision, you must first read the snapshot, and
bos@109: then every one of the revisions between the snapshot and your target
bos@109: revision.  The more history that a file accumulates, the more
bos@109: revisions you must read, hence the longer it takes to reconstruct a
bos@109: particular revision.
bos@109: 
bos@109: The innovation that Mercurial applies to this problem is simple but
bos@109: effective.  Once the cumulative amount of delta information stored
bos@109: since the last snapshot exceeds a fixed threshold, it stores a new
bos@109: snapshot (compressed, of course), instead of another delta.  This
bos@109: makes it possible to reconstruct \emph{any} revision of a file
bos@109: quickly.  This approach works so well that it has subsequently been
bos@109: copied by several other revision control systems.
bos@109: 
bos@109: \subsubsection{Aside: the influence of video compression}
bos@109: 
bos@109: If you're familiar with video compression or have ever watched a TV
bos@109: feed through a digital cable or satellite service, you may know that
bos@109: most video compression schemes store each frame of video as a delta
bos@109: against its predecessor frame.  In addition, these schemes use
bos@109: ``lossy'' compression techniques to increase the compression ratio, so
bos@109: visual errors accumulate over the course of a number of inter-frame
bos@109: deltas.
bos@109: 
bos@109: Because it's possible for a video stream to ``drop out'' occasionally
bos@109: due to signal glitches, and to limit the accumulation of artefacts
bos@109: introduced by the lossy compression process, video encoders
bos@109: periodically insert a complete frame (called a ``key frame'') into the
bos@109: video stream; the next delta is generated against that frame.  This
bos@109: means that if the video signal gets interrupted, it will resume once
bos@109: the next key frame is received.  Also, the accumulation of encoding
bos@109: errors restarts anew with each key frame.
bos@109: 
bos@109: \subsection{Clever compression}
bos@109: 
bos@109: When appropriate, Mercurial will store both snapshots and deltas in
bos@109: compressed form.  It does this by always \emph{trying to} compress a
bos@109: snapshot or delta, but only storing the compressed version if it's
bos@109: smaller than the uncompressed version.
bos@109: 
bos@109: This means that Mercurial does ``the right thing'' when storing a file
bos@109: whose native form is compressed, such as a \texttt{zip} archive or a
bos@109: JPEG image.  When these types of files are compressed a second time,
bos@109: the resulting file is usually bigger than the once-compressed form,
bos@109: and so Mercurial will store the plain \texttt{zip} or JPEG.
bos@109: 
bos@109: Deltas between revisions of a compressed file are usually larger than
bos@109: snapshots of the file, and Mercurial again does ``the right thing'' in
bos@109: these cases.  It finds that such a delta exceeds the threshold at
bos@109: which it should store a complete snapshot of the file, so it stores
bos@109: the snapshot, again saving space compared to a naive delta-only
bos@109: approach.
bos@109: 
bos@109: \subsection{Strong integrity}
bos@109: 
bos@109: Along with delta or snapshot information, a revlog entry contains a
bos@109: cryptographic hash of the data that it represents.  This makes it
bos@109: difficult to forge the contents of a revision, and easy to detect
bos@109: accidental corruption.
bos@109: 
bos@109: Mercurial checks these hashes when retrieving file revisions and when
bos@109: pulling changes from a repository.  If it encounters an integrity
bos@109: problem, it will complain and stop whatever it's doing.
bos@109: 
bos@109: In addition to the effect it has on retrieval efficiency, Mercurial's
bos@109: use of periodic snapshots makes it more robust against partial data
bos@109: corruption.  If a revlog becomes partly corrupted due to a hardware
bos@109: error or system bug, it's often possible to reconstruct some or most
bos@109: revisions from the uncorrupted sections of the revlog, both before and
bos@109: after the corrupted section.  This would not be possible with a
bos@109: delta-only storage model.
bos@109: 
bos@109: \subsection{Read/write ordering and atomicity}
bos@109: 
bos@109: Appending to files isn't the whole story when it comes to guaranteeing
bos@109: that a reader won't see a partial write.  If you recall
bos@109: figure~\ref{fig:concepts:metadata}, revisions in the changelog point to
bos@109: revisions in the manifest, and revisions in the manifest point to
bos@109: revisions in filelogs.  This hierarchy is deliberate.
bos@109: 
bos@109: A writer starts a transaction by writing filelog and manifest data,
bos@109: and doesn't write any changelog data until those are finished.  A
bos@109: reader starts by reading changelog data, then manifest data, followed
bos@109: by filelog data.
bos@109: 
bos@109: Since the writer has always finished writing filelog and manifest data
bos@109: before it writes to the changelog, a reader will never read a pointer
bos@109: to a partially written manifest revision from the changelog, and it will
bos@109: never read a pointer to a partially written filelog revision from the
bos@109: manifest.
bos@109: 
bos@109: \subsection{Concurrent access}
bos@109: 
bos@109: The read/write ordering and atomicity guarantees mean that Mercurial
bos@109: never needs to \emph{lock} a repository when it's reading data, even
bos@109: if the repository is being written to while the read is occurring.
bos@109: This has a big effect on scalability; you can have an arbitrary number
bos@109: of Mercurial processes safely reading data from a repository safely
bos@109: all at once, no matter whether it's being written to or not.
bos@109: 
bos@109: The lockless nature of reading means that if you're sharing a
bos@109: repository on a multi-user system, you don't need to grant other local
bos@109: users permission to \emph{write} to your repository in order for them
bos@109: to be able to clone it or pull changes from it; they only need
bos@109: \emph{read} permission.  (This is \emph{not} a common feature among
bos@109: revision control systems, so don't take it for granted!  Most require
bos@109: readers to be able to lock a repository to access it safely, and this
bos@109: requires write permission on at least one directory, which of course
bos@109: makes for all kinds of nasty and annoying security and administrative
bos@109: problems.)
bos@109: 
bos@109: Mercurial uses a locking mechanism to ensure that only one process can
bos@109: write to a repository at a time.  This locking mechanism is safe even
bos@109: over filesystems that are notoriously unsafe for locking, such as NFS.
bos@109: If a repository is locked, a writer will wait for a while to retry if
bos@109: the repository becomes unlocked, but if the repository remains locked
bos@109: for too long, the process attempting to write will time out after a
bos@109: while.  This means that your daily automated scripts won't get stuck
bos@109: forever and pile up if a system crashes unnoticed, for example.  (Yes,
bos@109: the timeout is configurable, from zero to infinity.)
bos@109: 
bos@109: 
bos@109: 
jeffpc@56: %%% Local Variables: 
jeffpc@56: %%% mode: latex
jeffpc@56: %%% TeX-master: "00book"
jeffpc@56: %%% End: