Scaling Git: Push Throughput

Write path for the MVCC backend on object storage: ingest on any pod, one committer per repository, one generation per batch.

This is a proposal under the Scaling Git design. It covers the write path only. Reads, routing, migration and the storage format are described by the parent document.

Summary

Scaling Git stores a repository as immutable, content-addressed artifacts in object storage. A manifest names the active set, and the latest generation object in a per-repository log names the active manifest. A write becomes durable when the writer creates the next generation object with a conditional put.

Concurrent pushes to one repository serialise on that put, and each of them merges, and uploads the merged manifest and reftable, inside the critical section. We propose to shrink the critical section to a single conditional put and to move everything else off the critical path:

  • Any Gitaly pod other than the repository’s committer ingests a push: it runs Git, uploads the push’s artifacts and computes a small delta of the reference changes and the packs added or retired.
  • One pod per repository, on which every pod agrees, is the committer. The PoC chooses it by rendezvous hashing; a control plane, as proposed for the parent’s clustering and routing, could make the choice instead. It folds deltas into an in-memory copy of the committed state and publishes one generation per batch.
  • The generation object the committer writes, the generation entry, carries the batch’s manifest and reftable beside the manifest hash. A commit is one write and no reads.
  • Repository maintenance and compaction run only on the committer. Garbage collection runs on any pod, against the generation log.

Motivation

In the parent design each write RPC resolves the head, runs Git, uploads the new artifacts and advances the pointer with a conditional put. Under concurrent pushes to one repository every push that loses the race re-reads the head, merges and tries again.

Grouping the waiting pushes so that one put publishes a whole batch removes the retries, but the pod holding the group still reads the head manifest and reftables from object storage, merges each push, writes and uploads a reftable and a manifest, then swaps the pointer. Throughput is bounded by that one pod and by several object-storage round trips per batch.

Goals

  • The per-push work of a hot repository, Git and the upload, spreads across ingest pods. Throughput grows with their number up to the committer’s fold rate: in-memory work per delta and one put per batch.
  • The work inside the critical section per batch is bounded and does not grow with the size of the repository.
  • Correctness continues to rest on the generation log alone, with no lock held outside the committer process.
  • Any pod can fail at any point of a push without a committed write being lost or applied twice. A push in flight at that moment either succeeds, after an internal retry if needed, or is reported to the client as failed. The artifacts of a push that never commits are reclaimed.

Non-Goals

  • Changes to Git. The maintenance policy and process-count changes on the Git side are a separate design document; this one relies on them where noted.
  • Routing of read traffic and cache placement. See the parent’s Clustering and routing.
  • Multi-region replication.
  • The retention policy for manifests. The parent’s Garbage Collection section owns it; this document adds to the mechanism only.

Assumptions

  • The parent’s assumptions about object storage hold.
  • Every pod agrees on the committer for a repository, whether computed from a shared view of the membership or assigned by a control plane. A brief disagreement during a change is tolerated.
  • Pods reach each other over gRPC.

Proposal

The write path is split into two roles that any Gitaly pod can hold, and the per-repository critical section is reduced to one object-storage write. A pod is one Gitaly process with its own cache; whether it runs alone on a machine or beside other pods does not matter to the design.

flowchart LR
  subgraph ingest[Ingest pods - any pod except the committer, up to the writer count]
    git[<b>Git</b><br/>receive-pack, index-pack, hooks]
    upload[<b>Upload</b><br/>pack, index, reftable]
    delta[<b>Delta</b><br/>reference records with old values,<br/>packs added or retired]
    git --> upload --> delta
  end

  subgraph committer[Committer pod - one per repository]
    queue[<b>Queue</b><br/>one per repository, drained in batches]
    fold[<b>Fold - the critical section</b><br/>in memory: references, pack set, reftable stack]
    unbundler[<b>Unbundler</b><br/>off the commit path]
    queue --> fold --> unbundler
  end

  subgraph os[Object storage - source of truth]
    artifacts[<b>Artifacts</b><br/>packs, indexes, reftables, manifests]
    log[<b>Generation log</b><br/>one entry per batch, put-if-absent,<br/>carrying the manifest and the batch's reftable]
  end

  upload -->|before the push joins the queue| artifacts
  delta -->|CommitMVCC| queue
  fold -->|one conditional put| log
  unbundler -->|manifest body and reftable<br/>under their own keys| artifacts
  • Ingest pod. Runs Git for the push (receive-pack, index-pack, the hooks), uploads the pack, index and reftable, computes the delta and sends it to the committer. A configuration value sets how many pods ingest for a repository. They are the next candidates of the same rendezvous ranking that picks the committer and the repository’s readers, with the committer excluded, so any pod ingests for some repositories and none for the ones it commits.
  • Committer. One pod per repository, on which every pod agrees; the PoC chooses it by rendezvous hashing over the cluster membership. It holds the repository’s committed state in memory, validates and folds deltas, writes one reftable per batch and creates the generation entry. It takes no pushes itself.
  • Unbundler. A worker pool on the committer’s pod, off the commit path. After each commit it puts the manifest body and the reftable the entry carries under their own keys, for the subsystems that read the bucket by key.
  • Object storage. The artifacts and the log are as in the parent. The generation object gains a second format, the generation entry: beside the manifest hash it carries the batch’s manifest body and reftable, and the origin and size of each artifact the batch added. Readers understand both formats.
  • Collector. Any pod. Reclaims artifacts that no retained generation references, following the parent’s Garbage Collection section with the additions below.

Design and implementation details

Ingest and the delta

The pod the router selected runs Git against its local cache as for any write RPC in the parent design. Before the push queues for commit it uploads the pack, index and reftable, computes a delta against the head its Git ran on (for each changed reference the old and new value, and the packs added or retired) and runs the pre-receive checks with the candidate manifest. A committer can apply the delta to any later head whose values for those references have not moved, which is Git’s own old-value rule for a push. The delta crosses the network in the internal CommitMVCC RPC.

The commit request

The request carries no bytes of the push. It names the head the push was built on and describes what changed.

Field Meaning
repository the target repository
base_generation, base_sha the head the session was built on, and the manifest it names
uploaded_sha the manifest the session’s upload published
request_id idempotency key; a retry after a lost response is answered from the first outcome
delta what changed, below; without it the session can only be committed as it is, when the head has not moved
origin_address the pod that holds the session’s artifacts in its cache, recorded in the generation entry so that peers can fetch from it

The delta has two halves. The reference half is a compare-and-set: the committer accepts a record when the running value equals the old value the session saw, or already equals the target. The object half is a set operation on the committed pack list.

Field Meaning
records the changed references at their new values, sorted by name; each record carries the target, the peeled object for an annotated tag, and whether it is symbolic or a deletion
old for each changed reference, the value the session saw it at; empty for one that did not exist
added_paths object artifacts the session added: packs with their indexes, the multi-pack index, the commit-graph
removed_paths object artifacts the session retired; the “retired twice” check runs against this list
pack_paths the session’s full pack list, present only when it retired packs or changed the multi-pack index or commit-graph
base_midx, base_graph, midx, graph the multi-pack index and commit-graph paths of the base and of the session, so the committer can settle which survive when both sides changed the pack set
ref_stack_paths, max_update_index, same_stack the session’s reftable stack and the update index its top table ends at; used only when a batch adopts the session’s manifest whole

A typical push is a handful of records, as many old values and two or three added paths. The large fields, pack_paths and ref_stack_paths, are populated only by repacks and adopt-whole commits.

The response carries the generation the publish landed in and the manifest it names, or a conflict: its kind (overlap or exhausted compare-and-swap), the generations and manifests involved, and the attempt count and backoff of the batch that rejected it, so the ingest pod can decide how to retry. The ingest pod holds the client connection until the committer answers. If the committer is unreachable or has moved, the pod re-sends once to the new committer, and otherwise commits through the conditional put itself.

The committer and its critical section

For each repository the committer keeps the committed state in memory: the references, the pack set and the reftable stack at the head it last published. Deltas for the repository are queued and drained in batches whose size grows with load.

flowchart LR
  subgraph ingest[On the ingest pod, per push]
    a[receive-pack, index-pack] --> b[upload pack, index, reftable] --> c[compute the delta, send it]
  end

  subgraph committer[On the committer, per batch, under the lock]
    d[check and apply each delta<br/>old values and pack rules, in memory] --> e[one reftable for the batch<br/>and the manifest, locally] --> f[one put-if-absent of the entry<br/>manifest and table inside]
  end

  c -->|CommitMVCC, batched with whatever is waiting| d

Under a per-repository lock the committer checks each delta against the running state: every old value still holds or the reference already has the target value, no retired pack is retired twice, and no reference is created under a live reference name. It applies the accepted deltas, writes one reftable for the batch and creates the next generation entry with a conditional put. The commit reads nothing from object storage and writes once per batch. A rejected delta is answered with a conflict, and the ingest pod retries it against the new head.

The committer trusts the head it wrote last; the conditional put remains as the safety net. During a membership change two pods may briefly both act as committer, the put serialises them, and the loser re-reads the head and folds its batch again. In steady state the put is almost never contested, so throughput is set by how many deltas fit in one round trip.

A delta re-sent after a lost response is applied at most once. A committer that answered the first copy answers the re-send from that outcome, keyed by request_id. A committer without that record folds the re-send like any other delta: a reference that still holds the delta’s target value passes the check; a reference a later push has moved on fails it, and so does a pack the first copy already retired. Such a re-send is answered with a conflict although its first copy committed. For a push that is the outcome-unknown row of the Failure modes table: the client is told the push failed, and its next push finds the repository up to date. For maintenance it is free: the pass re-reads the head after taking its lock. Recording the batch’s request ids in the generation entry would let a new committer answer such a re-send from the log; the design leaves that out.

Writers and membership

writer_replicas in the [elastic] configuration sets how many pods ingest for a repository. The committer is excluded from that set so that its CPU goes to folding. The router prefers the less loaded of two candidate ingest pods.

Batching and the generation entry

Everything in this section is orchestrator work. Git reads the result from its cache as in the parent: a manifest that names packs and a reftable stack.

In the parent design every artifact a generation names is durable before the generation object that names it, so a commit is several puts followed by one conditional put. The generation entry carries beside the manifest SHA the manifest body and the reftable the batch wrote, when the two together fit a size cap (1 MiB by default), so the commit is the one conditional put of the entry and a pod following the log receives the manifest and the table with the entry it reads anyway. Above the cap, as after a reftable compaction, they are put under their own keys ahead of the entry as in the parent. A log may hold generation objects of either format.

Every batch adds one table to the reftable stack. Once the stack is deeper than a configured depth, the committer compacts the segment that breaks Git’s geometric rule into one table, outside the critical section: the fold starts it in the background from a copy of the stack, it fetches and folds the segment’s tables, uploads the new table, and a later batch swaps the result in if the stack it was computed from is still a prefix of the current one. The segment is capped at what one reftable block holds; a larger segment, or a compaction that fails, leaves the stack as it is for Git to compact on its next direct write.

Manifest deltas

A repository with thousands of packs has a manifest of hundreds of kilobytes, which every pod serving the repository reads and parses with every generation, while the change between two generations is a few paths. From a configurable body size (64 KB by default) the entry carries the manifest as a delta to the previous generation’s: paths added, paths removed, and the reftable stack in order. A reader that holds the previous manifest applies the delta and checks the result against the SHA the entry names. Every 32nd generation carries the body whole, which bounds how far back a reader without a base walks the log.

Unbundling the entry

The unbundler puts the manifest body and the table the entry carries under their own keys after the commit, for the subsystems that read the bucket by key and never open an entry: a listing with metadata, a repository tool, the collector. It runs with bounded retries and is flushed on graceful shutdown; the commit does not wait for it. Until its puts land, a reader that misses an object under its key scans back from the head, within a bounded number of generations, for the entry that carries it. A committer that takes over a repository re-puts the objects carried by the entries inside that window before it serves, so a hole left by a failed committer is closed by its successor.

GITALY_MVCC_EMBED_GENERATIONS=false keeps the parent’s separate-upload path, so a cluster can be upgraded pod by pod with the readers first. Nothing in the on-disk Git format changes.

Artifacts served by peers

The pods that serve a repository are the top candidates of its rendezvous ranking, the reader set for reads and the writer set for pushes, so their number is set by configuration and does not grow with the cluster. Each of them downloads what a push wrote, so each push costs one object-store GET per artifact per serving pod, and the object store’s request rate becomes the first limit the cluster reaches.

The commit request names the pod that uploaded the delta’s artifacts (origin_address), and the committer records it in the generation entry as the origin of each artifact the generation added, with its size. A pod that needs an artifact it does not hold asks the origin first over the internal ReadMVCCArtifact RPC, which streams it from the origin’s local cache. The receiver checks the size and, for packs and indexes, the trailing checksum, and discards a mismatch. Whatever a peer cannot serve is fetched from the object store as before; artifacts are immutable and content-addressed, so what a peer serves under a path is what the object store holds under it.

A pod serves peers artifacts up to a size cap (4 MiB by default) and names itself as the origin of those alone. A push’s pack is small and wanted once by each other serving pod, which one pod serves easily. A repack’s merged pack is wanted by every serving pod at the same moment, which through one pod’s network interface is the pack size times the size of that set. Large artifacts therefore come from the object store, fetched in parallel parts.

Pre-warming large artifacts

When a generation names a new artifact above the peer-serving cap, in practice the pack a repack or compaction wrote, every serving pod downloads it before its next RPC on the repository can run, all at the moment the generation lands, which stalls every ingest pod’s pushes for as long as the download takes. The pod about to publish such an artifact already has it in the object store, so it asks every other serving pod to fetch it first, over the internal PrefetchMVCCArtifacts RPC, waits within a bound, and only then queues the publish. A push whose pack is below the cap, the common case, queues at once; one whose pack exceeds it waits for the same bounded round on its ingest pod before it queues. The round runs once per large artifact: repacks, compactions and the occasional large push. Pushes keep flowing meanwhile, and a peer that does not answer in time downloads at publish time as before.

Repository maintenance

Only the push RPCs fan out across the writer set. Every other mutator, maintenance among them, stays pinned to the rendezvous winner for the repository, which is the committer, so ingest pods never race a maintenance pass, and a pass commits what it wrote through the same queue as the pushes.

The committer never runs the Git that wrote the push packs, so without further work every pass would begin by fetching everything the pushes since its last pass added. The committer therefore hydrates every artifact a committed entry names with an origin into its own cache as the commit lands, from the origin pod and in the background; an artifact that fails to arrive is left to the pass’s own hydration. In the single-writer alternative below, the writer ran that Git itself and has nothing to hydrate.

A pass runs under a per-repository maintenance lock on the committer and reads the head after taking the lock, since a compaction may have committed meanwhile and a pass built on the older head would retire packs the head no longer names. GITALY_MVCC_PASS_MIN_INTERVAL sets the least time between two passes of one repository.

What a pass may do is bounded through Git configuration the pass sets: mvcc.maxRollupBytes caps what one geometric pass rolls up, mvcc.heldBackPackLimit how many packs it may hold back before the cap is raised, and mvcc.midxStaleObjects and mvcc.commitGraphStaleCommits how far the multi-pack index and the commit graph may fall behind before a pass rewrites them. Without the roll-up cap, each rewrite of the multi-pack index rolled the mid-sized packs into an ever larger one that every other pod then downloaded. The Git side of these options is specified in the separate Git design document.

Compaction of push packs

Every push is a pack, and every Git process and every maintenance pass pays for each live pack, so between passes the live pack count of a busy repository runs into the thousands. An opt-in compactor on the committer (GITALY_MVCC_COMPACT) merges the small packs that a few seconds of pushes added into one pack and publishes the manifest that names it. It is pack-objects over the new packs alone, in a scratch object directory that holds only them. It starts once a configured number of packs is pending or a configured interval has passed (GITALY_MVCC_COMPACT_MIN_PACKS, GITALY_MVCC_COMPACT_INTERVAL), and takes the same maintenance lock as a pass, since both retire the packs they merge. It is off by default: in our measurements the maintenance pass still did most of the merging, and full fetches were slower over the compacted packs.

Garbage collection

The parent’s Garbage Collection section specifies race-free pruning: a housekeeping pod lists manifests and artifacts, queues the unreferenced ones older than a threshold with their ETags, and deletes each conditionally on its ETag; a writer that starts to reference an artifact it did not upload refreshes that ETag first. This design keeps that mechanism and adds to it:

  • The generation log is the root set. The retained window is a range of generations, and the manifest of each is rebuilt from its entry, because a manifest can exist only in its entry until the unbundler puts it; a scan of the bucket’s manifest bodies alone would miss it and delete what it references. A delta entry is chained back to a whole body within a bounded number of entries.
  • An incremental tier that needs no listing. Each generation’s delta names the artifacts it retired, so a pass that replays the log from its last position finds the bulk of the garbage, the packs that maintenance and compaction retired. The parent’s listing pass remains for the uploads of pushes that were never committed.
  • The collector runs on any pod. It reads the log and the bucket and touches neither the pointer nor the committer’s state. A missed pass leaves garbage a little longer.
  • The threshold is a promise to readers. It must exceed the longest time an uncommitted or superseded artifact can be held by a push in flight, a peer prefetch or a reader pinned to an old manifest. We derive the RPC deadline from it.
  • Deletes are paced. Compaction retires packs at the rate of pushes, so a burst of deletes against one prefix would throttle the pushes that share it.

Truncating the generation log itself is open. The commit path relies on the log being contiguous, so the parent’s retention policy applies above a floor the commit path can read.

Failure modes

The push path has one durable step, the conditional put. Everything before it can be repeated and everything after it can be recovered from the entry. Publish runs before Git reports the status to the client and before post-receive, so a client is never told that a push succeeded before its generation exists.

Failure What the client sees Durable state
Ingest pod fails before its uploads push fails nothing written
Ingest pod fails after its uploads, before the commit request push fails unreferenced artifacts; the collector reclaims them
Ingest pod fails after sending the commit request push fails may be committed; the client’s next push finds it up to date; post-receive did not run
Committer fails before the batch’s put the ingest pod re-sends to the new committer, or commits itself the uploaded artifacts
Committer fails after the put, before answering the re-sent delta folds as a no-op and the push succeeds; if a later push moved the references first, the re-send is rejected and the client sees a failed push committed once
Committer’s put times out with the outcome unknown push fails may be committed; the client’s next push finds it up to date; post-receive did not run; the entry was not handed to the unbundler
Committer fails after answering, before the unbundler has put the entry’s objects push succeeds the batch’s manifest and reftable exist in the entry only
Two committers during a membership change slower the conditional put serializes them; the loser re-folds

Alternative Solutions

Per-push conditional put

The parent’s write lifecycle, sufficient for repositories with little write concurrency. Under concurrent pushes it degrades into retries on the pointer. This design is a superset of it: a repository with one writer at a time still commits one generation per push.

A single writer per repository

Routing every write for a repository to one pod, which the clustering and routing proposal allows with a writer count of one, gives the batching in this design without the ingest split: the writer queues the pushes it receives, folds them in memory and publishes one generation per batch. The commit request, the delta format and the ingest-pod fallback are not needed, and the writer’s cache already holds what it wrote, so maintenance needs no hydration.

What remains per push on that pod is Git itself, receive-pack, index-pack and the hooks, and the upload of the pack. Those costs do not batch, so the one writer’s CPU, memory and network are the ceiling for a hot repository. This design turns the single writer into the committer: it stops taking pushes, which move to separate ingest pods placed in front of it, while its queue, fold and generation entry stay as they are. A deployment can therefore start with the single writer and add ingest pods for the repositories whose push rate outgrows one pod.

Transactional writes to object storage

Object storage has no multi-key transaction, and each way of approximating one costs more than collecting. Staging uploads under a pending prefix and promoting them on commit adds a copy per artifact inside the window this design reduces to one put. Deleting on rejection is unsafe with content-addressed de-duplication, because a second push may already reference the object. A pod that fails between upload and commit runs no compensation at all, and no write-time scheme reclaims the packs that maintenance retires, which are most of the garbage.

A durable, out-of-process commit queue

The committer’s queue holds nothing durable: each item is a live request from an ingest pod whose artifacts are already in object storage and which still holds the client connection, and if the committer disappears the ingest pod re-sends or commits itself. Git promises no order between concurrent pushes, so losing arrival order costs a re-fold. A durable queue would add a write per push on the hot path to protect state that is already durable and already retried.

Garbage collection on the committer

The committer knows what each batch retired, but so does the generation log. Running the collector on the committer would put listing and delete I/O on the one pod the push path depends on, for no information the log does not already carry.

Further reading