Aspen: execution plan

Aspen is a self-hostable Git server, written in Rust, built for agent-scale traffic. It follows the design OpenAI published for Pando ("Scaling Git at OpenAI", https://openai.com/index/scaling-git-at-openai/, 2026-11-02):

• packs are stored in object storage without being unpacked; a thin pushed pack is first completed into a self-contained pack (spike S3);
• branch history lives in a database, and each branch updates independently;
• a push sees only the trunk branch;
• fetches are served by selecting stored packs, never by building one per client;
• small packs are merged in the background in levels, like an LSM tree.

It also takes inspiration from Cursor's "Git at any scale" (https://cursor.com/blog/git-at-any-scale, August 2026): push throughput is measured against Cursor's published numbers, counting a push only once it is durable, and group commit batches the write that publishes pushes the way Cursor's design does (decisions log, "Group commit"). README.md, "Where the design comes from", says which idea came from which source.

Aspen is open source under the Apache License 2.0 and published at https://github.com/reason-machines/aspen by a regular sync from the maintainers' development repository. Write everything (code, docs, commit history) as if strangers will read it. This file is the source of truth for scope and sequencing. Update it when a spike or a review changes a decision, and record why.

Owner: Aspen, a coding agent that coordinates the work: it keeps this plan, reviews and merges every pull request, and runs each node in its own agent session (the "Agent session" column below). Requested by the maintainers, 2026-11-02.

Progress

The coordinator updates this table after every merge. States: ready, in progress, in review, merged, blocked. Node IDs are defined in section 3.

---

1. Product

"A Git server that stays fast when a thousand agents push at once, and that you can run yourself on any cloud in five minutes."

Target users: teams whose agents, CI and humans hammer one repository. They run Aspen in one of two ways:

• Primary mode: Aspen is the Git host.
• Mirror mode: Aspen sits in front of GitHub or GitLab. Fetches come from Aspen's own packs, and pushes are forwarded upstream either synchronously (proxy) or in the background (Aspen is primary, upstream is a downstream mirror).

Mirror mode is the main adoption path, because nobody has to migrate.

Goals

• Works with stock git, over HTTP and SSH, with no client changes required.
• Pushes to different branches scale horizontally. The only contended write is a compare-and-swap on one branch's tip.
• Reads scale to the network limit. Clients that set git config fetch.uriprotocols https download packs straight from object storage or a CDN.
• Runs anywhere: one binary, Postgres, and any S3-compatible bucket (S3, GCS, Azure Blob, R2, MinIO, or the local filesystem).
• Reliability is proven by differential testing against stock Git, chaos testing and fuzzing, not asserted.

Not in v1 (document each one, and give a clear error where needed)

• Partial-clone filters: don't advertise filter, so clients fall back to a full fetch (verified in S5). Lazy fetches from existing partial clones (want <blob-oid>) get a clear ERR; mirror mode may forward them upstream later.
• Exact shallow packs for every deepen form: a plain --depth=N fetch of a ref version's tip (N up to 64) is served an exact pack, built once per version and depth (node C8). --deepen (relative), --shallow-since, --shallow-exclude, --unshallow, deeper depths and non-tip wants are still served the full history of the wants plus an exact shallow-info (S5). Operators can set shallow = refuse to answer with a clear ERR instead, or ASPEN_SHALLOW_PACKS=false to serve every shallow fetch the full history.
• SHA-256 repositories.
• A web UI for browsing code.
• Code review or issues.

---

2. Architecture

git client ─HTTP/SSH─► aspen-server ─CAS per ref─► Postgres
                            │                      (refs, ref_versions, commit_index, covers)
                            │ stream+validate
                            ▼
                     object store (S3/GCS/Azure/R2/MinIO/fs) ◄── aspen-compactor (L0→L4 covers)
                            ▲
git client ◄─packfile-uri── aspen-express (regional pack cache) / CDN

Data model (schema v1, N1)

The tables below are schema v1, in crates/aspen-meta/migrations (Postgres and SQLite, kept identical) and described column by column in crates/aspen-meta/SCHEMA.md. The Rust contracts are in crates/aspen-core. Object IDs are 20-byte binary, timestamps are microseconds since the epoch, and every cross-table reference inside a repository is a composite foreign key through repo_id.

Push path

Confirmed against git 2.30, 2.39, 2.56 and OpenAI's fork by spike S2. The full rule table and the decisive wire excerpts are in bench/spikes/s2/RESULT.md.

1. The client connects and Aspen runs v0/v1 receive-pack. It advertises trunk as the only named ref, plus earlier trunk tips as .have lines, or capabilities^{} for an empty repository: every one of the newest 8 × have_tips trunk versions (64 by default), then tips about a quarter apart down to the oldest trunk version a push searches (meta::have_tip_seqs; at most 108 lines for a million versions). Without the .have lines, a client that has not fetched the latest trunk pushes its whole history; with them, a client any number of landings behind resends at most about a quarter of them (decisions log, "item 10"). Capabilities: report-status delete-refs side-band-64k quiet atomic ofs-delta push-options object-format=sha1 agent.

   • A branch-scoped URL, /<repo>.git/-/refs/heads/<branch>, also advertises that one ref. It is the opt-in route for stock --force-with-lease and git push --delete <branch>.
   • push.negotiate. Through the plain URL, stock Git resends every commit of the pushed branch that is not on trunk. With git config push.negotiate true it first runs a negotiation-only v2 fetch (wait-for-done, haves, no wants); Aspen advertises wait-for-done and acknowledges every have with a commit_index row, and Git leaves those commits out. A fast-forward of the pushed branch then needs nothing beyond its usual dependencies (issue #203's branch dependency). A push whose left-out commits are outside its dependencies (another branch's commits) is completed from their versions (decisions log, "push.negotiate").

2. Ingest produces a self-contained pack, which packfile URIs require (spike S3), and never unpacks it. It is synchronous and takes two passes:

   • Pass 1 (receive): the incoming pack streams to a spool file on local disk (a real disk, never tmpfs) while gix-pack parses it. It inflates every entry, checks each entry's CRC32, and checks the client's trailer with SHA-1 collision detection. Memory stays under 10 MiB at any pack size; the parse runs at about 110 MB/s per core (S4). The base IDs of all REF_DELTA entries are collected as candidate bases. A candidate may be a thin base (in deps) or an object in the same pack: libgit2 sends every delta as REF_DELTA, mostly against bases in the pack (S1's push_order.py), and pass 1 cannot tell the two apart, because a deltified entry's ID is known only once it is resolved. A push whose header object count exceeds the limit is refused before its body is read.
   • Base prefetch: each candidate, with its delta chain, is looked up in parallel before completion starts. The base-object lookup probes the .idx of the covering packs of every ref version advertised on the connection: trunk, plus the branch for a branch-scoped URL, which are exactly the push's deps (see S3's README). A candidate that deps doesn't have is not an error: it is expected to be in the pack. Prefetching matters because gix asks for bases one at a time from inside its parse. A lookup that went to the blob store there would serialize thousands of round trips per push: S4 counted 5,431 entry reads for a 100-commit Linux push. The prefetch is designed, not measured: S4's harness used a synchronous lookup over a local pack, and B1 measures the prefetch against a real bucket. Every stored pack's delta chains are known before they are read (chain maps from its entry table: in memory beside a whole .idx, on the spool disk beside a sampled one), so all the entries of every candidate's chain are read in one round of parallel range reads, whatever the chain's depth (decisions log, item 11).
   • Completion and index: gix-pack re-reads the spool with a lookup that answers only from the prefetched bases. For each REF_DELTA whose base the lookup returns, gix inserts the base as a full object just before that delta and turns the delta into an OFS_DELTA. So every completion base comes before the entries that use it (S1's base-first rule); placing all of them in front of the received entries, as S1 first proposed, is not needed. A REF_DELTA the lookup has no answer for stays a REF_DELTA and gix resolves it inside the pack, in either direction: gix accepts a base that comes later in the pack. gix then rewrites the header count, re-hashes the pack (about 1 GB/s), and resolves every delta in parallel. That computes each object ID, again with SHA-1 collision detection, and writes the .idx. A REF_DELTA whose base is in neither deps nor the pack fails the push with unpack error. Every push takes this path: for a pack that needs no bases it costs 4% more CPU and returns the same bytes (S4).
   • No object twice. gix asks the lookup about every REF_DELTA base it did not insert itself, without checking whether the pack already holds that object. So when a pack carries object X plus a REF_DELTA against X, and deps also has X, completion stores X twice. S4 reproduced this (dup.sh): a full-history resend written all-REF_DELTA, as libgit2 writes deltas, came out with 92 entries for 90 objects, and git index-pack --strict rejected it. The rule: after indexing, one linear scan of the sorted .idx looks for an ID that appears twice. If any does, completion runs again over the spool with a lookup that answers "not found" for those IDs, so their deltas resolve against the copy the client sent, and the scan runs again. On the same pack that gives 90 unique objects, accepted by --strict. Filtering the lookup during pass 1 is not enough on its own: pass 1 can hash full objects but not deltified ones, so a deltified X would still be inserted twice. The scan is needed anyway as the guarantee, and the rerun is a rare path (a client resending objects the server has, with REF_DELTA against them).
   • A pack that itself holds an object twice is rejected. If the scan after the rerun still finds a repeated ID, the client sent both copies, and no lookup answer can remove either. The push fails with unpack error naming the duplicated object, and no packs row is written. This is a deliberate difference from stock git receive-pack, which accepts such a pack unless --strict (or receive.fsckObjects) is on. No tested client sends one (S1, S4), and storing it would break the no-duplicates invariant the stitcher relies on.
   • Forward bases in the received entries are repaired, never rejected (S1). A client's own pack may put a REF_DELTA before the entry it is based on; stock git receive-pack accepts that, so rejecting it would break pushes that work everywhere else. gix stores such a pack as it came, with the delta before its base, which would later trip the fetch path's forward-base ERR and its page. So the repair runs after completion, the duplicate check and the .idx, on gix's output:
     • Detection. Only an entry that is still a REF_DELTA can point forward (OFS_DELTA distances always point back). The skip decision uses the final completion run: if its lookup answered every candidate (no prefetch miss, and no ID refused by a duplicate rerun), no REF_DELTA remains and the check is skipped; this is every stock git push, which sends OFS_DELTA. A duplicate rerun refuses IDs on purpose, so the REF_DELTAs against them stay and can point forward; after a rerun the check always runs. Otherwise a header walk of the completed pack (entry boundaries from the sorted .idx offsets, no inflation) compares each remaining REF_DELTA's offset with its base's offset from the .idx.
     • Repair. If a base comes later, the pack is rewritten on local disk: entries keep their order, except that each entry whose base comes later (and, transitively, any entry based on it) moves to just after its base. Compressed data is copied unchanged; OFS_DELTA headers whose distance changed, and the CRC32 of each changed entry, are recomputed. The new pack is re-hashed (about 1 GB/s) and its .idx rebuilt from the known IDs, without resolving deltas again. S1 found no client that sends forward bases: git 2.30–2.56, JGit, libgit2 and go-git all put every base first in 32 pushes (push_order.py). So the repair is a rare path that has to be correct, not a fast one. Neither the header walk nor the rewrite has been measured.
   • Connectivity check, as git receive-pack does. Only objects that the incoming pack references (commit parents and trees, tree entries, tag targets) but does not contain are checked. Each must exist in the push's dependency closure, found with the same base-object lookup, batched. Nothing reachable from those objects is walked: every committed ref version is connected (see ref_versions), so existence implies connectivity. A missing object fails the push.
   • Pass 2 (upload): a multi-ref push may store more than one pack: B1 places each write's objects in trunk's pack, its own pack or one shared pack (decisions log, "where a write's objects are stored"). For each stored pack, one sequential read feeds a multipart upload, and the .idx and the entry table (fetch path step 4) are uploaded beside it. The trailer of each stored pack becomes its packs.checksum. Each pack is registered before its upload, which goes to the keys of the returned row (decisions log, C1 pack generations).
   • Only once the pack and .idx are durable may step 4 run. Every packs row is self-contained, for every reader, with no flag.

   The trunk version is searched because its .have tips are trunk ancestors; the scoped branch because the client built its pack against it too. The per-push probe cost bound is in bench/spikes/s2/RESULT.md.

   Thin packs stay accepted on the wire; no-thin is an optional per-repository advertisement (S2).

   Costs that S4 measured on Linux, a 6.55 GB pack with 12.0 M objects (bench/spikes/s4/README.md):

   • CPU per push, the whole pipeline: pass 1 (64 CPU-seconds with the spool) plus completion and index over the spool (282), about 346 CPU-seconds, against 680 for git index-pack, which does both in one process. The duplicate scan is one pass over the .idx. Each rare path adds its own cost: a duplicate rerun repeats the 282, and a forward-base repair adds a local copy and a re-hash. Prefetch I/O and the upload come on top and are measured by B1.
   • Memory: about 140 bytes of heap per object (1.6 GiB), plus the mmapped pack, which is reclaimable page cache.
   • Disk: about twice the push, plus its .idx.
   • Stored size: completion grows packs most for small pushes, from 320 B to 37 kB for one commit and from 10.5 MB to 57 MB for 100 commits. Git's --fix-thin grows them the same way.

   B1 measures push latency end to end, against a real bucket.

   Test obligations for step 2. A2 covers each with a unit test on constructed packs, and B1 covers each end to end through git push and the libgit2 client in A6's matrix. Each stored pack must index under stock git index-pack without --fix-thin, and no ID may repeat in its .idx (--strict also rejects a repeat, but it checks connectivity too, so it only fits a test pack that is closed):

   • an all-REF_DELTA pack whose bases are in the pack and not in deps (an ordinary libgit2 push): every candidate misses in deps and the push is accepted;
   • an all-REF_DELTA resend of objects deps already has, including a REF_DELTA against a deltified object of the pack (S4's dup.sh is the model): no duplicate is stored;
   • a REF_DELTA before its in-pack base, with and without that base also in deps: stored base-first, no duplicate;
   • a REF_DELTA whose base is in neither the pack nor deps: unpack error, and no packs row;
   • a received pack that holds the same object twice: unpack error naming that object, and no packs row.

3. The commit check runs. The client never saw any branch except trunk, so (full table in RESULT.md):

   • Create: accepted.
   • Existing branch, zero old-oid (every update of an unadvertised branch looks like this): Aspen runs the fast-forward check itself against commit_index, falling back to the ref version's packs for commits not indexed yet (see the commit_index row). If ancestry can't be decided, the update gets a retryable ng, never a guessed verdict. If new equals the current tip, it is a no-op.
   • Non-fast-forward: git push --force is invisible to the server. Accept only with push option -o force (or force=<ref>) or a matching -o lease=<ref>:<oid>, and only where allow_force permits.
   • Delete: clients refuse --delete <short-name> for unadvertised branches. --delete refs/heads/<branch> arrives with a zero old-oid, so it is unconditional: like a force, it needs -o force and allow_force. A delete with a real old-oid (branch-scoped URL, or a lease) is a compare-and-swap delete and needs no force. Trunk cannot be deleted.
   • Tags never fast-forward: replacing one needs -o force.
   • Exact replay: a command refused only because the ref already moved can still be ok. It is when the ref's current tip is the command's new value (or it is deleted, for a delete), the version just before it had the command's old value (or its lease's), and the command's own rules (lease, force signal and policy, fast-forward, tag and trunk protection), checked against that previous version, accept it: answer ok and write nothing. libcurl resends a receive-pack POST when a reused connection dies before any answer, so the first copy can land with its answer lost (#132). This matches the outcome, not the request: another push that made the same move, and that this command would also have been allowed to make, counts too. Only the current version's parent counts; if the ref moved again, the parent cannot be read, or the command's rules refuse the move, the command keeps its refusal. Creates take no part: a zero old value claims nothing, so a resent create is checked like any zero-old command (a plain no-op while the ref is still at its value).
   • Atomic: all commands in one transaction, or all rejected.
   • Rejection reasons go in the ng line and on side-band 2, even under quiet (every non-terminal client asks for it).

   Every verdict (fast-forward, lease, delete, no-op) is computed against one exact ref_version of the target ref, and that version is the one the step 4 compare-and-swap expects.

4. One transaction inserts the ref_version, sets the refs row with compare-and-swap, and enqueues jobs. That makes the push read-your-own-write with O(1) metadata reads and writes. If the compare-and-swap loses because the ref moved, Aspen recomputes the step 3 verdict against the new version (a bounded number of times), or returns a retryable ng. It never commits a verdict computed against a stale version. A3's and B1's tests, and the differential suite, cover two concurrent pushes to one branch.

   The pack's trunk dependency, recorded in deps, is the exact trunk version that step 2's lookup and connectivity check searched. It is never re-read at commit time: where a repository allows trunk force, trunk can be rewound between step 2 and this step, and a re-read would record a version whose packs were never searched. Smart HTTP does not tell the server which trunk tip it advertised, so a pack built against a rewound tip is caught by that connectivity check, not assumed away; trunk being fast-forward-only by default only makes this rare.

   A forced trunk update (only with allow_force = all) is the one exception, fixed at push time by B1 (decisions log): its trunk_dep is the newest trunk version whose tip is an ancestor of the new tip, its pack also holds the objects the new tip reaches that are only in the skipped versions' closures, and orphan history stores its full closure with no trunk dependency. The new version is then exact.

Fetch path (protocol v2)

1. ls-refs honours ref-prefix, so the server never sends a 100 MB ref list. Each instance keeps a repository's refs in memory once it has listed them, and every later ls-refs asks the store only which refs changed since (MetaStore::ref_changes, one statement over ref_versions.txid), so polls by stock git ls-remote <url> <ref>, which sends no ref-prefix, cost one statement and no ref rows. A listing is never stale: each request is answered from a store read that began after it arrived (decisions log, "ls-refs snapshots").

2. fetch maps the client's wants to ref versions through commit_index, and each have to a version only when that version's closure is exact and its tip is the have or an ancestor of it (fetch::HaveVersion); other haves are dropped, which only costs bytes. The planner then picks the covering packs, using the cover levels as a skip list, so the pack count grows logarithmically with the versions sent. On the have side the walk skips through chained covers (C1): when it expands a held version at seq = s that lies inside a cover carrying a chain, every version of that ref from the cover's first to s is held (each depends on its parent, so its closure contains the earlier ones'), and the walk jumps to the cover's first version instead of stepping down one version at a time. A client current on trunk fetching a branch created N trunk pushes ago visits a handful of versions per cover level plus the uncompacted tail, not N. The want side is still linear: a full clone visits every version of its closure. B2 exports PlanStats.visited as a metric; PlanStats.chain_skips (how many cover chains the walk jumped through) is in the planner's API for its tests and for D2, which benchmarks the stale-branch fetch (section 4). A cover without a chain never changes what the walk treats as held (decisions log, cover-subtraction rule and "C1: cover chains").

3. Clients that set fetch.uriprotocols get an empty inline pack plus packfile-uris that point at presigned URLs or the CDN. Rules from spike S3 (bench/spikes/s3/README.md):

   • the advertised hash is the pack's trailing SHA-1 (packs.checksum);
   • only URIs whose scheme the client listed (clients do not filter);
   • always a packfile section, even when it is empty;
   • URLs are direct (no redirect), self-authorizing and honour Range;
   • direct-to-bucket URIs go only to clients whose agent= parses as git/2.56 or later. Older opted-in clients get URIs only on an endpoint with the 206 rule below: aspen-express, or a CDN tier declared ASPEN_CDN_PAST_END=206. A CDN tier declared 416 (the default) is for git/2.56 and later only. With no such endpoint they get the stitched pack. The agent string is self-reported, so this is a reliability choice, not security;
   • each planned pack goes out as an express or CDN URI first, then as the store's own URL, then stitched into the inline pack (decisions log, "C2: delivery order per planned pack");
   • an Aspen-served blob endpoint answers a range at or past the end with 206 and an empty body, not 416 (git ≤ 2.55 gets stuck on 416);
   • advertise no-ref-delta only when every planned pack is REF_DELTA-free (open for B2: over smart HTTP the advertisement precedes the request, so B2 sets the policy; see the decisions log);
   • soft cap on URIs per response, with a stitched pack beyond it.

4. Every other client gets a stitched pack, built from the selected packs' entries in plan order:

   • drop every entry whose object id was already emitted (keep the first copy);
   • copy every other entry's compressed data unchanged; rewrite only the OFS_DELTA base distance, against wherever the base copy was emitted;
   • write the header object count (known from the plan before the first byte) and recompute the trailing SHA-1.

   Invariants of every stitched pack, enforced by A2 and its fuzz target: each object id at most once, every delta's base earlier in the pack, and never thin (stored packs are self-contained, and go-git never asks for thin-pack). A2 also measures the maximum delta chain depth of stitched packs: keep-first can hang a delta in a later segment off an earlier segment's copy of its base that is itself a delta, so depth can approach the sum of the segments' depths. A2 checks that git 2.30 index-pack accepts the worst case the compaction levels allow. Spike S1 (bench/spikes/s1/) showed why: plain concatenation is accepted by every client only when no object repeats. Overlapping covers and completed push bases do repeat objects, and then libgit2 rejects the pack, git 2.30–2.47 reject it when the repeated object is a REF_DELTA base, every git rejects it with fsckObjects, and go-git writes a pack stock git can't read. Naive deduplication can make delta cycles that exhaust the client's memory. With base-first stored packs (packs invariant), keep-first deduplication emits every base before its deltas in a single pass.