jerojasro@343: \chapter{Behind the scenes} jerojasro@343: \label{chap:concepts} jerojasro@343: jerojasro@343: Unlike many revision control systems, the concepts upon which jerojasro@343: Mercurial is built are simple enough that it's easy to understand how jerojasro@343: the software really works. Knowing this certainly isn't necessary, jerojasro@343: but I find it useful to have a ``mental model'' of what's going on. jerojasro@343: jerojasro@343: This understanding gives me confidence that Mercurial has been jerojasro@343: carefully designed to be both \emph{safe} and \emph{efficient}. And jerojasro@343: just as importantly, if it's easy for me to retain a good idea of what jerojasro@343: the software is doing when I perform a revision control task, I'm less jerojasro@343: likely to be surprised by its behaviour. jerojasro@343: jerojasro@343: In this chapter, we'll initially cover the core concepts behind jerojasro@343: Mercurial's design, then continue to discuss some of the interesting jerojasro@343: details of its implementation. jerojasro@343: jerojasro@343: \section{Mercurial's historical record} jerojasro@343: jerojasro@343: \subsection{Tracking the history of a single file} jerojasro@343: jerojasro@343: When Mercurial tracks modifications to a file, it stores the history jerojasro@343: of that file in a metadata object called a \emph{filelog}. Each entry jerojasro@343: in the filelog contains enough information to reconstruct one revision jerojasro@343: of the file that is being tracked. Filelogs are stored as files in jerojasro@343: the \sdirname{.hg/store/data} directory. A filelog contains two kinds jerojasro@343: of information: revision data, and an index to help Mercurial to find jerojasro@343: a revision efficiently. jerojasro@343: jerojasro@343: A file that is large, or has a lot of history, has its filelog stored jerojasro@343: in separate data (``\texttt{.d}'' suffix) and index (``\texttt{.i}'' jerojasro@343: suffix) files. For small files without much history, the revision jerojasro@343: data and index are combined in a single ``\texttt{.i}'' file. The jerojasro@343: correspondence between a file in the working directory and the filelog jerojasro@343: that tracks its history in the repository is illustrated in jerojasro@343: figure~\ref{fig:concepts:filelog}. jerojasro@343: jerojasro@343: \begin{figure}[ht] jerojasro@343: \centering jerojasro@343: \grafix{filelog} jerojasro@343: \caption{Relationships between files in working directory and jerojasro@343: filelogs in repository} jerojasro@343: \label{fig:concepts:filelog} jerojasro@343: \end{figure} jerojasro@343: jerojasro@343: \subsection{Managing tracked files} jerojasro@343: jerojasro@343: Mercurial uses a structure called a \emph{manifest} to collect jerojasro@343: together information about the files that it tracks. Each entry in jerojasro@343: the manifest contains information about the files present in a single jerojasro@343: changeset. An entry records which files are present in the changeset, jerojasro@343: the revision of each file, and a few other pieces of file metadata. jerojasro@343: jerojasro@343: \subsection{Recording changeset information} jerojasro@343: jerojasro@343: The \emph{changelog} contains information about each changeset. Each jerojasro@343: revision records who committed a change, the changeset comment, other jerojasro@343: pieces of changeset-related information, and the revision of the jerojasro@343: manifest to use. jerojasro@343: jerojasro@343: \subsection{Relationships between revisions} jerojasro@343: jerojasro@343: Within a changelog, a manifest, or a filelog, each revision stores a jerojasro@343: pointer to its immediate parent (or to its two parents, if it's a jerojasro@343: merge revision). As I mentioned above, there are also relationships jerojasro@343: between revisions \emph{across} these structures, and they are jerojasro@343: hierarchical in nature. jerojasro@343: jerojasro@343: For every changeset in a repository, there is exactly one revision jerojasro@343: stored in the changelog. Each revision of the changelog contains a jerojasro@343: pointer to a single revision of the manifest. A revision of the jerojasro@343: manifest stores a pointer to a single revision of each filelog tracked jerojasro@343: when that changeset was created. These relationships are illustrated jerojasro@343: in figure~\ref{fig:concepts:metadata}. jerojasro@343: jerojasro@343: \begin{figure}[ht] jerojasro@343: \centering jerojasro@343: \grafix{metadata} jerojasro@343: \caption{Metadata relationships} jerojasro@343: \label{fig:concepts:metadata} jerojasro@343: \end{figure} jerojasro@343: jerojasro@343: As the illustration shows, there is \emph{not} a ``one to one'' jerojasro@343: relationship between revisions in the changelog, manifest, or filelog. jerojasro@343: If the manifest hasn't changed between two changesets, the changelog jerojasro@343: entries for those changesets will point to the same revision of the jerojasro@343: manifest. If a file that Mercurial tracks hasn't changed between two jerojasro@343: changesets, the entry for that file in the two revisions of the jerojasro@343: manifest will point to the same revision of its filelog. jerojasro@343: jerojasro@343: \section{Safe, efficient storage} jerojasro@343: jerojasro@343: The underpinnings of changelogs, manifests, and filelogs are provided jerojasro@343: by a single structure called the \emph{revlog}. jerojasro@343: jerojasro@343: \subsection{Efficient storage} jerojasro@343: jerojasro@343: The revlog provides efficient storage of revisions using a jerojasro@343: \emph{delta} mechanism. Instead of storing a complete copy of a file jerojasro@343: for each revision, it stores the changes needed to transform an older jerojasro@343: revision into the new revision. For many kinds of file data, these jerojasro@343: deltas are typically a fraction of a percent of the size of a full jerojasro@343: copy of a file. jerojasro@343: jerojasro@343: Some obsolete revision control systems can only work with deltas of jerojasro@343: text files. They must either store binary files as complete snapshots jerojasro@343: or encoded into a text representation, both of which are wasteful jerojasro@343: approaches. Mercurial can efficiently handle deltas of files with jerojasro@343: arbitrary binary contents; it doesn't need to treat text as special. jerojasro@343: jerojasro@343: \subsection{Safe operation} jerojasro@343: \label{sec:concepts:txn} jerojasro@343: jerojasro@343: Mercurial only ever \emph{appends} data to the end of a revlog file. jerojasro@343: It never modifies a section of a file after it has written it. This jerojasro@343: is both more robust and efficient than schemes that need to modify or jerojasro@343: rewrite data. jerojasro@343: jerojasro@343: In addition, Mercurial treats every write as part of a jerojasro@343: \emph{transaction} that can span a number of files. A transaction is jerojasro@343: \emph{atomic}: either the entire transaction succeeds and its effects jerojasro@343: are all visible to readers in one go, or the whole thing is undone. jerojasro@343: This guarantee of atomicity means that if you're running two copies of jerojasro@343: Mercurial, where one is reading data and one is writing it, the reader jerojasro@343: will never see a partially written result that might confuse it. jerojasro@343: jerojasro@343: The fact that Mercurial only appends to files makes it easier to jerojasro@343: provide this transactional guarantee. The easier it is to do stuff jerojasro@343: like this, the more confident you should be that it's done correctly. jerojasro@343: jerojasro@343: \subsection{Fast retrieval} jerojasro@343: jerojasro@343: Mercurial cleverly avoids a pitfall common to all earlier jerojasro@343: revision control systems: the problem of \emph{inefficient retrieval}. jerojasro@343: Most revision control systems store the contents of a revision as an jerojasro@343: incremental series of modifications against a ``snapshot''. To jerojasro@343: reconstruct a specific revision, you must first read the snapshot, and jerojasro@343: then every one of the revisions between the snapshot and your target jerojasro@343: revision. The more history that a file accumulates, the more jerojasro@343: revisions you must read, hence the longer it takes to reconstruct a jerojasro@343: particular revision. jerojasro@343: jerojasro@343: \begin{figure}[ht] jerojasro@343: \centering jerojasro@343: \grafix{snapshot} jerojasro@343: \caption{Snapshot of a revlog, with incremental deltas} jerojasro@343: \label{fig:concepts:snapshot} jerojasro@343: \end{figure} jerojasro@343: jerojasro@343: The innovation that Mercurial applies to this problem is simple but jerojasro@343: effective. Once the cumulative amount of delta information stored jerojasro@343: since the last snapshot exceeds a fixed threshold, it stores a new jerojasro@343: snapshot (compressed, of course), instead of another delta. This jerojasro@343: makes it possible to reconstruct \emph{any} revision of a file jerojasro@343: quickly. This approach works so well that it has since been copied by jerojasro@343: several other revision control systems. jerojasro@343: jerojasro@343: Figure~\ref{fig:concepts:snapshot} illustrates the idea. In an entry jerojasro@343: in a revlog's index file, Mercurial stores the range of entries from jerojasro@343: the data file that it must read to reconstruct a particular revision. jerojasro@343: jerojasro@343: \subsubsection{Aside: the influence of video compression} jerojasro@343: jerojasro@343: If you're familiar with video compression or have ever watched a TV jerojasro@343: feed through a digital cable or satellite service, you may know that jerojasro@343: most video compression schemes store each frame of video as a delta jerojasro@343: against its predecessor frame. In addition, these schemes use jerojasro@343: ``lossy'' compression techniques to increase the compression ratio, so jerojasro@343: visual errors accumulate over the course of a number of inter-frame jerojasro@343: deltas. jerojasro@343: jerojasro@343: Because it's possible for a video stream to ``drop out'' occasionally jerojasro@343: due to signal glitches, and to limit the accumulation of artefacts jerojasro@343: introduced by the lossy compression process, video encoders jerojasro@343: periodically insert a complete frame (called a ``key frame'') into the jerojasro@343: video stream; the next delta is generated against that frame. This jerojasro@343: means that if the video signal gets interrupted, it will resume once jerojasro@343: the next key frame is received. Also, the accumulation of encoding jerojasro@343: errors restarts anew with each key frame. jerojasro@343: jerojasro@343: \subsection{Identification and strong integrity} jerojasro@343: jerojasro@343: Along with delta or snapshot information, a revlog entry contains a jerojasro@343: cryptographic hash of the data that it represents. This makes it jerojasro@343: difficult to forge the contents of a revision, and easy to detect jerojasro@343: accidental corruption. jerojasro@343: jerojasro@343: Hashes provide more than a mere check against corruption; they are jerojasro@343: used as the identifiers for revisions. The changeset identification jerojasro@343: hashes that you see as an end user are from revisions of the jerojasro@343: changelog. Although filelogs and the manifest also use hashes, jerojasro@343: Mercurial only uses these behind the scenes. jerojasro@343: jerojasro@343: Mercurial verifies that hashes are correct when it retrieves file jerojasro@343: revisions and when it pulls changes from another repository. If it jerojasro@343: encounters an integrity problem, it will complain and stop whatever jerojasro@343: it's doing. jerojasro@343: jerojasro@343: In addition to the effect it has on retrieval efficiency, Mercurial's jerojasro@343: use of periodic snapshots makes it more robust against partial data jerojasro@343: corruption. If a revlog becomes partly corrupted due to a hardware jerojasro@343: error or system bug, it's often possible to reconstruct some or most jerojasro@343: revisions from the uncorrupted sections of the revlog, both before and jerojasro@343: after the corrupted section. This would not be possible with a jerojasro@343: delta-only storage model. jerojasro@343: jerojasro@343: \section{Revision history, branching, jerojasro@343: and merging} jerojasro@343: jerojasro@343: Every entry in a Mercurial revlog knows the identity of its immediate jerojasro@343: ancestor revision, usually referred to as its \emph{parent}. In fact, jerojasro@343: a revision contains room for not one parent, but two. Mercurial uses jerojasro@343: a special hash, called the ``null ID'', to represent the idea ``there jerojasro@343: is no parent here''. This hash is simply a string of zeroes. jerojasro@343: jerojasro@343: In figure~\ref{fig:concepts:revlog}, you can see an example of the jerojasro@343: conceptual structure of a revlog. Filelogs, manifests, and changelogs jerojasro@343: all have this same structure; they differ only in the kind of data jerojasro@343: stored in each delta or snapshot. jerojasro@343: jerojasro@343: The first revision in a revlog (at the bottom of the image) has the jerojasro@343: null ID in both of its parent slots. For a ``normal'' revision, its jerojasro@343: first parent slot contains the ID of its parent revision, and its jerojasro@343: second contains the null ID, indicating that the revision has only one jerojasro@343: real parent. Any two revisions that have the same parent ID are jerojasro@343: branches. A revision that represents a merge between branches has two jerojasro@343: normal revision IDs in its parent slots. jerojasro@343: jerojasro@343: \begin{figure}[ht] jerojasro@343: \centering jerojasro@343: \grafix{revlog} jerojasro@343: \caption{} jerojasro@343: \label{fig:concepts:revlog} jerojasro@343: \end{figure} jerojasro@343: jerojasro@343: \section{The working directory} jerojasro@343: jerojasro@343: In the working directory, Mercurial stores a snapshot of the files jerojasro@343: from the repository as of a particular changeset. jerojasro@343: jerojasro@343: The working directory ``knows'' which changeset it contains. When you jerojasro@343: update the working directory to contain a particular changeset, jerojasro@343: Mercurial looks up the appropriate revision of the manifest to find jerojasro@343: out which files it was tracking at the time that changeset was jerojasro@343: committed, and which revision of each file was then current. It then jerojasro@343: recreates a copy of each of those files, with the same contents it had jerojasro@343: when the changeset was committed. jerojasro@343: jerojasro@343: The \emph{dirstate} contains Mercurial's knowledge of the working jerojasro@343: directory. This details which changeset the working directory is jerojasro@343: updated to, and all of the files that Mercurial is tracking in the jerojasro@343: working directory. jerojasro@343: jerojasro@343: Just as a revision of a revlog has room for two parents, so that it jerojasro@343: can represent either a normal revision (with one parent) or a merge of jerojasro@343: two earlier revisions, the dirstate has slots for two parents. When jerojasro@343: you use the \hgcmd{update} command, the changeset that you update to jerojasro@343: is stored in the ``first parent'' slot, and the null ID in the second. jerojasro@343: When you \hgcmd{merge} with another changeset, the first parent jerojasro@343: remains unchanged, and the second parent is filled in with the jerojasro@343: changeset you're merging with. The \hgcmd{parents} command tells you jerojasro@343: what the parents of the dirstate are. jerojasro@343: jerojasro@343: \subsection{What happens when you commit} jerojasro@343: jerojasro@343: The dirstate stores parent information for more than just book-keeping jerojasro@343: purposes. Mercurial uses the parents of the dirstate as \emph{the jerojasro@343: parents of a new changeset} when you perform a commit. jerojasro@343: jerojasro@343: \begin{figure}[ht] jerojasro@343: \centering jerojasro@343: \grafix{wdir} jerojasro@343: \caption{The working directory can have two parents} jerojasro@343: \label{fig:concepts:wdir} jerojasro@343: \end{figure} jerojasro@343: jerojasro@343: Figure~\ref{fig:concepts:wdir} shows the normal state of the working jerojasro@343: directory, where it has a single changeset as parent. That changeset jerojasro@343: is the \emph{tip}, the newest changeset in the repository that has no jerojasro@343: children. jerojasro@343: jerojasro@343: \begin{figure}[ht] jerojasro@343: \centering jerojasro@343: \grafix{wdir-after-commit} jerojasro@343: \caption{The working directory gains new parents after a commit} jerojasro@343: \label{fig:concepts:wdir-after-commit} jerojasro@343: \end{figure} jerojasro@343: jerojasro@343: It's useful to think of the working directory as ``the changeset I'm jerojasro@343: about to commit''. Any files that you tell Mercurial that you've jerojasro@343: added, removed, renamed, or copied will be reflected in that jerojasro@343: changeset, as will modifications to any files that Mercurial is jerojasro@343: already tracking; the new changeset will have the parents of the jerojasro@343: working directory as its parents. jerojasro@343: jerojasro@343: After a commit, Mercurial will update the parents of the working jerojasro@343: directory, so that the first parent is the ID of the new changeset, jerojasro@343: and the second is the null ID. This is shown in jerojasro@343: figure~\ref{fig:concepts:wdir-after-commit}. Mercurial doesn't touch jerojasro@343: any of the files in the working directory when you commit; it just jerojasro@343: modifies the dirstate to note its new parents. jerojasro@343: jerojasro@343: \subsection{Creating a new head} jerojasro@343: jerojasro@343: It's perfectly normal to update the working directory to a changeset jerojasro@343: other than the current tip. For example, you might want to know what jerojasro@343: your project looked like last Tuesday, or you could be looking through jerojasro@343: changesets to see which one introduced a bug. In cases like this, the jerojasro@343: natural thing to do is update the working directory to the changeset jerojasro@343: you're interested in, and then examine the files in the working jerojasro@343: directory directly to see their contents as they werea when you jerojasro@343: committed that changeset. The effect of this is shown in jerojasro@343: figure~\ref{fig:concepts:wdir-pre-branch}. jerojasro@343: jerojasro@343: \begin{figure}[ht] jerojasro@343: \centering jerojasro@343: \grafix{wdir-pre-branch} jerojasro@343: \caption{The working directory, updated to an older changeset} jerojasro@343: \label{fig:concepts:wdir-pre-branch} jerojasro@343: \end{figure} jerojasro@343: jerojasro@343: Having updated the working directory to an older changeset, what jerojasro@343: happens if you make some changes, and then commit? Mercurial behaves jerojasro@343: in the same way as I outlined above. The parents of the working jerojasro@343: directory become the parents of the new changeset. This new changeset jerojasro@343: has no children, so it becomes the new tip. And the repository now jerojasro@343: contains two changesets that have no children; we call these jerojasro@343: \emph{heads}. You can see the structure that this creates in jerojasro@343: figure~\ref{fig:concepts:wdir-branch}. jerojasro@343: jerojasro@343: \begin{figure}[ht] jerojasro@343: \centering jerojasro@343: \grafix{wdir-branch} jerojasro@343: \caption{After a commit made while synced to an older changeset} jerojasro@343: \label{fig:concepts:wdir-branch} jerojasro@343: \end{figure} jerojasro@343: jerojasro@343: \begin{note} jerojasro@343: If you're new to Mercurial, you should keep in mind a common jerojasro@343: ``error'', which is to use the \hgcmd{pull} command without any jerojasro@343: options. By default, the \hgcmd{pull} command \emph{does not} jerojasro@343: update the working directory, so you'll bring new changesets into jerojasro@343: your repository, but the working directory will stay synced at the jerojasro@343: same changeset as before the pull. If you make some changes and jerojasro@343: commit afterwards, you'll thus create a new head, because your jerojasro@343: working directory isn't synced to whatever the current tip is. jerojasro@343: jerojasro@343: I put the word ``error'' in quotes because all that you need to do jerojasro@343: to rectify this situation is \hgcmd{merge}, then \hgcmd{commit}. In jerojasro@343: other words, this almost never has negative consequences; it just jerojasro@343: surprises people. I'll discuss other ways to avoid this behaviour, jerojasro@343: and why Mercurial behaves in this initially surprising way, later jerojasro@343: on. jerojasro@343: \end{note} jerojasro@343: jerojasro@343: \subsection{Merging heads} jerojasro@343: jerojasro@343: When you run the \hgcmd{merge} command, Mercurial leaves the first jerojasro@343: parent of the working directory unchanged, and sets the second parent jerojasro@343: to the changeset you're merging with, as shown in jerojasro@343: figure~\ref{fig:concepts:wdir-merge}. jerojasro@343: jerojasro@343: \begin{figure}[ht] jerojasro@343: \centering jerojasro@343: \grafix{wdir-merge} jerojasro@343: \caption{Merging two heads} jerojasro@343: \label{fig:concepts:wdir-merge} jerojasro@343: \end{figure} jerojasro@343: jerojasro@343: Mercurial also has to modify the working directory, to merge the files jerojasro@343: managed in the two changesets. Simplified a little, the merging jerojasro@343: process goes like this, for every file in the manifests of both jerojasro@343: changesets. jerojasro@343: \begin{itemize} jerojasro@343: \item If neither changeset has modified a file, do nothing with that jerojasro@343: file. jerojasro@343: \item If one changeset has modified a file, and the other hasn't, jerojasro@343: create the modified copy of the file in the working directory. jerojasro@343: \item If one changeset has removed a file, and the other hasn't (or jerojasro@343: has also deleted it), delete the file from the working directory. jerojasro@343: \item If one changeset has removed a file, but the other has modified jerojasro@343: the file, ask the user what to do: keep the modified file, or remove jerojasro@343: it? jerojasro@343: \item If both changesets have modified a file, invoke an external jerojasro@343: merge program to choose the new contents for the merged file. This jerojasro@343: may require input from the user. jerojasro@343: \item If one changeset has modified a file, and the other has renamed jerojasro@343: or copied the file, make sure that the changes follow the new name jerojasro@343: of the file. jerojasro@343: \end{itemize} jerojasro@343: There are more details---merging has plenty of corner cases---but jerojasro@343: these are the most common choices that are involved in a merge. As jerojasro@343: you can see, most cases are completely automatic, and indeed most jerojasro@343: merges finish automatically, without requiring your input to resolve jerojasro@343: any conflicts. jerojasro@343: jerojasro@343: When you're thinking about what happens when you commit after a merge, jerojasro@343: once again the working directory is ``the changeset I'm about to jerojasro@343: commit''. After the \hgcmd{merge} command completes, the working jerojasro@343: directory has two parents; these will become the parents of the new jerojasro@343: changeset. jerojasro@343: jerojasro@343: Mercurial lets you perform multiple merges, but you must commit the jerojasro@343: results of each individual merge as you go. This is necessary because jerojasro@343: Mercurial only tracks two parents for both revisions and the working jerojasro@343: directory. While it would be technically possible to merge multiple jerojasro@343: changesets at once, the prospect of user confusion and making a jerojasro@343: terrible mess of a merge immediately becomes overwhelming. jerojasro@343: jerojasro@343: \section{Other interesting design features} jerojasro@343: jerojasro@343: In the sections above, I've tried to highlight some of the most jerojasro@343: important aspects of Mercurial's design, to illustrate that it pays jerojasro@343: careful attention to reliability and performance. However, the jerojasro@343: attention to detail doesn't stop there. There are a number of other jerojasro@343: aspects of Mercurial's construction that I personally find jerojasro@343: interesting. I'll detail a few of them here, separate from the ``big jerojasro@343: ticket'' items above, so that if you're interested, you can gain a jerojasro@343: better idea of the amount of thinking that goes into a well-designed jerojasro@343: system. jerojasro@343: jerojasro@343: \subsection{Clever compression} jerojasro@343: jerojasro@343: When appropriate, Mercurial will store both snapshots and deltas in jerojasro@343: compressed form. It does this by always \emph{trying to} compress a jerojasro@343: snapshot or delta, but only storing the compressed version if it's jerojasro@343: smaller than the uncompressed version. jerojasro@343: jerojasro@343: This means that Mercurial does ``the right thing'' when storing a file jerojasro@343: whose native form is compressed, such as a \texttt{zip} archive or a jerojasro@343: JPEG image. When these types of files are compressed a second time, jerojasro@343: the resulting file is usually bigger than the once-compressed form, jerojasro@343: and so Mercurial will store the plain \texttt{zip} or JPEG. jerojasro@343: jerojasro@343: Deltas between revisions of a compressed file are usually larger than jerojasro@343: snapshots of the file, and Mercurial again does ``the right thing'' in jerojasro@343: these cases. It finds that such a delta exceeds the threshold at jerojasro@343: which it should store a complete snapshot of the file, so it stores jerojasro@343: the snapshot, again saving space compared to a naive delta-only jerojasro@343: approach. jerojasro@343: jerojasro@343: \subsubsection{Network recompression} jerojasro@343: jerojasro@343: When storing revisions on disk, Mercurial uses the ``deflate'' jerojasro@343: compression algorithm (the same one used by the popular \texttt{zip} jerojasro@343: archive format), which balances good speed with a respectable jerojasro@343: compression ratio. However, when transmitting revision data over a jerojasro@343: network connection, Mercurial uncompresses the compressed revision jerojasro@343: data. jerojasro@343: jerojasro@343: If the connection is over HTTP, Mercurial recompresses the entire jerojasro@343: stream of data using a compression algorithm that gives a better jerojasro@343: compression ratio (the Burrows-Wheeler algorithm from the widely used jerojasro@343: \texttt{bzip2} compression package). This combination of algorithm jerojasro@343: and compression of the entire stream (instead of a revision at a time) jerojasro@343: substantially reduces the number of bytes to be transferred, yielding jerojasro@343: better network performance over almost all kinds of network. jerojasro@343: jerojasro@343: (If the connection is over \command{ssh}, Mercurial \emph{doesn't} jerojasro@343: recompress the stream, because \command{ssh} can already do this jerojasro@343: itself.) jerojasro@343: jerojasro@343: \subsection{Read/write ordering and atomicity} jerojasro@343: jerojasro@343: Appending to files isn't the whole story when it comes to guaranteeing jerojasro@343: that a reader won't see a partial write. If you recall jerojasro@343: figure~\ref{fig:concepts:metadata}, revisions in the changelog point to jerojasro@343: revisions in the manifest, and revisions in the manifest point to jerojasro@343: revisions in filelogs. This hierarchy is deliberate. jerojasro@343: jerojasro@343: A writer starts a transaction by writing filelog and manifest data, jerojasro@343: and doesn't write any changelog data until those are finished. A jerojasro@343: reader starts by reading changelog data, then manifest data, followed jerojasro@343: by filelog data. jerojasro@343: jerojasro@343: Since the writer has always finished writing filelog and manifest data jerojasro@343: before it writes to the changelog, a reader will never read a pointer jerojasro@343: to a partially written manifest revision from the changelog, and it will jerojasro@343: never read a pointer to a partially written filelog revision from the jerojasro@343: manifest. jerojasro@343: jerojasro@343: \subsection{Concurrent access} jerojasro@343: jerojasro@343: The read/write ordering and atomicity guarantees mean that Mercurial jerojasro@343: never needs to \emph{lock} a repository when it's reading data, even jerojasro@343: if the repository is being written to while the read is occurring. jerojasro@343: This has a big effect on scalability; you can have an arbitrary number jerojasro@343: of Mercurial processes safely reading data from a repository safely jerojasro@343: all at once, no matter whether it's being written to or not. jerojasro@343: jerojasro@343: The lockless nature of reading means that if you're sharing a jerojasro@343: repository on a multi-user system, you don't need to grant other local jerojasro@343: users permission to \emph{write} to your repository in order for them jerojasro@343: to be able to clone it or pull changes from it; they only need jerojasro@343: \emph{read} permission. (This is \emph{not} a common feature among jerojasro@343: revision control systems, so don't take it for granted! Most require jerojasro@343: readers to be able to lock a repository to access it safely, and this jerojasro@343: requires write permission on at least one directory, which of course jerojasro@343: makes for all kinds of nasty and annoying security and administrative jerojasro@343: problems.) jerojasro@343: jerojasro@343: Mercurial uses locks to ensure that only one process can write to a jerojasro@343: repository at a time (the locking mechanism is safe even over jerojasro@343: filesystems that are notoriously hostile to locking, such as NFS). If jerojasro@343: a repository is locked, a writer will wait for a while to retry if the jerojasro@343: repository becomes unlocked, but if the repository remains locked for jerojasro@343: too long, the process attempting to write will time out after a while. jerojasro@343: This means that your daily automated scripts won't get stuck forever jerojasro@343: and pile up if a system crashes unnoticed, for example. (Yes, the jerojasro@343: timeout is configurable, from zero to infinity.) jerojasro@343: jerojasro@343: \subsubsection{Safe dirstate access} jerojasro@343: jerojasro@343: As with revision data, Mercurial doesn't take a lock to read the jerojasro@343: dirstate file; it does acquire a lock to write it. To avoid the jerojasro@343: possibility of reading a partially written copy of the dirstate file, jerojasro@343: Mercurial writes to a file with a unique name in the same directory as jerojasro@343: the dirstate file, then renames the temporary file atomically to jerojasro@343: \filename{dirstate}. The file named \filename{dirstate} is thus jerojasro@343: guaranteed to be complete, not partially written. jerojasro@343: jerojasro@343: \subsection{Avoiding seeks} jerojasro@343: jerojasro@343: Critical to Mercurial's performance is the avoidance of seeks of the jerojasro@343: disk head, since any seek is far more expensive than even a jerojasro@343: comparatively large read operation. jerojasro@343: jerojasro@343: This is why, for example, the dirstate is stored in a single file. If jerojasro@343: there were a dirstate file per directory that Mercurial tracked, the jerojasro@343: disk would seek once per directory. Instead, Mercurial reads the jerojasro@343: entire single dirstate file in one step. jerojasro@343: jerojasro@343: Mercurial also uses a ``copy on write'' scheme when cloning a jerojasro@343: repository on local storage. Instead of copying every revlog file jerojasro@343: from the old repository into the new repository, it makes a ``hard jerojasro@343: link'', which is a shorthand way to say ``these two names point to the jerojasro@343: same file''. When Mercurial is about to write to one of a revlog's jerojasro@343: files, it checks to see if the number of names pointing at the file is jerojasro@343: greater than one. If it is, more than one repository is using the jerojasro@343: file, so Mercurial makes a new copy of the file that is private to jerojasro@343: this repository. jerojasro@343: jerojasro@343: A few revision control developers have pointed out that this idea of jerojasro@343: making a complete private copy of a file is not very efficient in its jerojasro@343: use of storage. While this is true, storage is cheap, and this method jerojasro@343: gives the highest performance while deferring most book-keeping to the jerojasro@343: operating system. An alternative scheme would most likely reduce jerojasro@343: performance and increase the complexity of the software, each of which jerojasro@343: is much more important to the ``feel'' of day-to-day use. jerojasro@343: jerojasro@343: \subsection{Other contents of the dirstate} jerojasro@343: jerojasro@343: Because Mercurial doesn't force you to tell it when you're modifying a jerojasro@343: file, it uses the dirstate to store some extra information so it can jerojasro@343: determine efficiently whether you have modified a file. For each file jerojasro@343: in the working directory, it stores the time that it last modified the jerojasro@343: file itself, and the size of the file at that time. jerojasro@343: jerojasro@343: When you explicitly \hgcmd{add}, \hgcmd{remove}, \hgcmd{rename} or jerojasro@343: \hgcmd{copy} files, Mercurial updates the dirstate so that it knows jerojasro@343: what to do with those files when you commit. jerojasro@343: jerojasro@343: When Mercurial is checking the states of files in the working jerojasro@343: directory, it first checks a file's modification time. If that has jerojasro@343: not changed, the file must not have been modified. If the file's size jerojasro@343: has changed, the file must have been modified. If the modification jerojasro@343: time has changed, but the size has not, only then does Mercurial need jerojasro@343: to read the actual contents of the file to see if they've changed. jerojasro@343: Storing these few extra pieces of information dramatically reduces the jerojasro@343: amount of data that Mercurial needs to read, which yields large jerojasro@343: performance improvements compared to other revision control systems. jerojasro@343: jerojasro@343: %%% Local Variables: jerojasro@343: %%% mode: latex jerojasro@343: %%% TeX-master: "00book" jerojasro@343: %%% End: