Scion Source Control Specification

Status: Draft v1 (core system; language-agnostic contracts, reference stack noted)

Purpose: Define a server-centric, always-online source control system for a single large monorepo used concurrently by humans, coding agents, and CI.

Provenance: This is a clean-room specification. Its only external design input is the public article "Redesigning source control" (anthropic.com/engineering/redesigning-source-control). All formats, schemas, algorithms, and thresholds below are original to this document. Contributors MUST NOT introduce material derived from non-public implementations of similar systems.

Normative Language

The key words MUST, MUST NOT, REQUIRED, SHOULD, SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL in this document are to be interpreted as described in RFC 2119.

Implementation-defined means the behavior is part of the implementation contract, but this specification does not prescribe one universal policy. Implementations MUST document the selected behavior.

Deferred marks behavior intentionally out of scope for v1. Every deferred item is listed in Appendix A, and the v1 design MUST NOT preclude it (see Appendix C).

1. Problem Statement

Scion stores and serves the history of one monorepo whose history and working copies are too large, and whose write rate is too high, for a decentralized full-clone system to serve efficiently.

The system solves five problems:

• Clients get the files they need without cloning history: every read is a request for an immutable, content-addressed object that any cache can answer.
• Commits, rebases, merges, and landings run on the server, cost proportional to the size of the change (not the repository), and require no global lock or merge queue.
• History queries (log, blame, diff, ancestry) are server APIs, so tools and CI jobs need no local repository.
• Working copies are optional and sparse; a client materializes only the directories it asks for.
• Content that must not exist (leaked secrets, legally required removals) can be removed without rewriting history or changing any identifier.

Important boundaries:

• Scion is the source of truth for repository content, history, and refs.
• Code review, code search, and CI orchestration are separate systems that consume Scion APIs. The only review-related concept in Scion is the stable ChangeId (Section 4.3.5).
• Semantic correctness of main (does it build, do tests pass) is enforced by post-merge and pre-deploy testing outside Scion. Scion guarantees textual merge correctness only.

2. Goals and Non-Goals

2.1 Goals

• One repository, many concurrent writers (humans, agents, CI), Unix clients (Linux, macOS).
• Immutable, content-addressed object model with self-verifying identifiers.
• Changesets stored as deltas against their first parent; trees computed server-side in O(changed paths x depth).
• Lock-free landing onto refs via rebase + single-row compare-and-swap, with idempotent retries.
• A generic, versioned derived-data framework so new history indexes need no client changes.
• Two read tiers (per-node daemon, authoritative server) speaking identical read RPCs.
• Sparse materialized working copies with a correct, stat-cache-based status.
• GitHub login for humans; configured service principals for agents and CI.
• Path-prefix access control defined in a config file and applied retroactively.
• Redaction of file contents and changeset metadata without history rewrite.
• One-way git import for migration and differential testing.

2.2 Non-Goals

• Offline operation, peer-to-peer sync, or a full local clone.
• Git wire-protocol or on-disk compatibility.
• A global merge queue or any landing serialization beyond the ref CAS.
• Semantic (build/test) conflict detection.
• Multi-repository hosting in v1 (the schema is repo-keyed so it can be added; Appendix C).
• Code review UI, code search, CI scheduling.
• Windows clients.

2.3 Deferred (designed-for, not built in v1)

Regional cache tier, virtualized (FUSE/NFS) working copies, filesystem-watcher status, daemon snapshot bootstrap, chunked large-blob storage, rename/copy detection, path-name redaction, garbage collection of unreferenced objects, multi-repo, two-way git sync, search index, land policies (review/CI gates). Details: Appendix A.

3. System Overview

3.1 Main Components

1. Object Store

   • S3-compatible object storage holding zstd-compressed object bytes.
   • Holds all blobs, and trees/metadata objects larger than the inline threshold.

2. Metadata Store

   • CockroachDB cluster (SERIALIZABLE isolation).
   • Holds object index, changesets, refs, ref log, landed-change index, derived data, path index, redactions, and counters.

3. scion-server (authoritative tier; stateless apart from caches; horizontally scalable)

   • Object Service: serves and accepts objects; enforces integrity and redaction.
   • Write Service: validates and creates changesets; builds trees.
   • Merge Engine: deterministic rebase and three-way merge.
   • Ref Service: ref resolution, CAS updates, land loop, ref log.
   • Derivation Engine: eager and lazy derived data.
   • Query Service: log, blame, diff, read-file, ancestry, merge-base.
   • Auth Service: GitHub and service-principal login; token issuance.
   • Policy Engine: loads acl.toml; evaluates path and ref permissions.
   • Redaction Service: redaction lifecycle, propagation, purge.

4. scion-daemon (per-node caching tier; OPTIONAL per node)

   • Listens on a local Unix socket.
   • Serves read RPCs from a bounded, verified, content-addressed cache; forwards everything else.
   • Shared by all jobs and users on the node.

5. scion client (CLI + library)

   • Manages working copies, sparse profiles, the stat cache, and local operation state.
   • Talks to a daemon socket or directly to a server; the protocol is identical.

6. scion-import

   • One-way git-to-Scion history importer (Section 14).

3.2 Abstraction Levels

1. Object Layer: canonical encoding, identifiers, blobs, trees, metadata, changesets.
2. Storage Layer: metadata schema, object storage layout, write ordering.
3. History Layer: changeset creation, rebase/merge, refs, land.
4. Derived Layer: commit graph, history nodes, blame, path index.
5. Service Layer: RPC contracts, error model, authentication, authorization.
6. Cache Layer: daemon cache, coalescing, epoch-driven invalidation.
7. Working Copy Layer: checkout, sparse, status, commit, conflict resolution.
8. Policy Layer: acl.toml, service principals, redaction authority.

Lower layers MUST NOT depend on higher layers. In particular, the object and storage layers MUST NOT depend on policy (authorization is applied in the service layer).

3.3 Topology and Request Flow

 client ──unix socket──▶ scion-daemon ──TLS gRPC──▶ scion-server ──▶ CockroachDB
   │                        (cache)                     │       └──▶ Object storage
   └────────────TLS gRPC (no daemon)───────────────────┘

• Read RPCs are answered by the nearest tier holding the object; misses go one hop upstream.
• Write, ref, query, auth, and admin RPCs are forwarded by the daemon verbatim.
• A tier MUST talk only to the tier directly above it. The topology is invisible to clients.
• A future regional cache tier inserts between daemon and server without protocol changes.

3.4 Reference Stack

• Implementation language: Rust for client, daemon, and server.
• RPC: gRPC with protobuf message envelopes (protobuf is used for transport only; object identity uses the canonical encoding in Section 4.1, never protobuf serialization).
• Hash: BLAKE3-256. Compression: zstd.
• Metadata: CockroachDB. Objects: S3-compatible storage (MinIO in tests).

3.5 External Dependencies

• CockroachDB v23.2 or later.
• S3-compatible object storage with strong read-after-write consistency.
• GitHub OAuth App (device flow) and the GitHub REST API (GET /user).
• Unix host for clients and daemons.

4. Core Domain Model

4.1 Canonical Encoding

All structured objects use one canonical binary encoding. Identity is defined over these bytes.

Primitives:

• u8: one byte.
• uvarint: unsigned LEB128, at most 10 bytes, MUST be minimally encoded.
• svarint: zigzag-mapped uvarint for signed 64-bit values.
• bytes: uvarint length followed by that many octets.
• list<T>: uvarint count followed by T repeated.
• option<T>: u8 0 (absent) or 1 (present) followed by T when present.
• ObjectId: 34 raw bytes (Section 4.2).
• Extensions: list<(tag: uvarint, value: bytes)> with strictly ascending tags; always the final field of a structured object.

Decoders MUST reject: non-minimal varints, lengths exceeding remaining input, trailing bytes, lists that violate their ordering or uniqueness rules, unknown format_version values, and unknown extension tags that are odd (critical). Unknown even (ignorable) extension tags MUST be preserved byte-for-byte and otherwise ignored. Encoders MUST produce exactly one byte string per logical value.

4.2 Object Identifiers

ObjectId = algo (u8) || kind (u8) || digest (32 bytes)

• algo: 0x01 = BLAKE3-256. Other values reserved.
• kind: 0x01 Blob, 0x02 Tree, 0x03 Changeset, 0x04 Meta. 0x10-0x7F reserved for derived objects (Section 9). Other values reserved.
• digest:
  • Blob: BLAKE3 of the raw (uncompressed) content.
  • All other kinds: BLAKE3 of the canonical encoding.

Text form: kind letter (b, t, c, m; derived kinds x), algo as one decimal digit, then 64 lowercase hex digits. Example: c1 + 64 hex. Clients MAY display and accept unique prefixes of at least 8 hex digits for changesets.

Because kind is part of the identifier, two objects of different kinds never collide even when their digests are equal.

4.3 Entities

4.3.1 Blob

• Raw file bytes (regular file, executable file, or symlink target).
• Stored zstd-compressed; compression level and framing are not part of identity.
• Size limit: limits.max_blob_bytes (default 104857600). Hard v1 ceiling: 4 GiB.

4.3.2 Tree

Tree {
  format_version: uvarint = 1
  entries: list<TreeEntry>     // strictly ascending by name, bytewise unsigned compare
  extensions: Extensions
}
TreeEntry {
  name: bytes
  mode: u8                     // 1 File, 2 Executable, 3 Symlink, 4 Tree
  id: ObjectId                 // kind Blob for modes 1-3, kind Tree for mode 4
  size: uvarint                // blob length for modes 1-3; 0 for mode 4
}

• Modes correspond to git modes 100644, 100755, 120000, and 040000.
• A non-root tree MUST have at least one entry (no empty directories, matching git).
• The root tree MAY be empty.

4.3.3 Meta

Meta {
  format_version: uvarint = 1
  author: bytes                // conventionally "Name <email>"
  author_time: Timestamp
  committer: option<bytes>     // populated by import; absent for native changesets
  committer_time: option<Timestamp>
  creator: bytes               // principal id that created the changeset (server-set)
  message: bytes
  extensions: Extensions       // tag 2 (ignorable): git commit id from import
}
Timestamp { seconds: svarint, tz_offset_minutes: svarint }

Meta is a separate object so that it can be redacted independently (Section 13) without changing any changeset identifier.

4.3.4 Changeset

Changeset {
  format_version: uvarint = 1
  parents: list<ObjectId>      // kind Changeset; ordered; unique; 0..=limits.max_parents
  root_tree: ObjectId          // kind Tree
  meta: ObjectId               // kind Meta
  change_id: 16 bytes
  delta: list<DeltaEntry>      // strictly ascending by path, bytewise
  extensions: Extensions
}
DeltaEntry {
  path: bytes
  op: u8                       // 1 Put, 2 Delete
  // present only when op = Put:
  mode: u8                     // 1..=3
  id: ObjectId                 // kind Blob
  size: uvarint
}

Let T1 be the root tree of parents[0], or the empty tree when parents is empty. A changeset is valid only if:

• delta equals FileDiff(T1, root_tree) exactly: the set of file-level paths whose (mode, id) differs, with Put for present-in-result and Delete for absent-in-result. Consequently the delta is uniquely determined by the two trees, is minimal, and contains no no-op entries.
• Exactly one changeset per repository has empty parents: the root changeset with an empty root_tree. (Import may create others; see Section 14.)

delta is relative to the first parent only, including for merge changesets.

4.3.5 ChangeId

• 16 random bytes generated by the client when a logical change is first committed.
• Preserved by amend and rebase; identifies "the same change" across revisions for review tools and for land idempotency.
• Not unique per changeset. Unique per ref among landed changes (Section 8.4).
• Text form: 32 lowercase hex digits.

4.3.6 Ref

• (name, target: ChangesetId, version: u64).
• Name grammar: ^[a-z0-9][a-z0-9._/-]{0,199}$, no .., no //, no trailing /.
• v1 refs: main (REQUIRED), release/*, green/* (last-known-good, written by CI). Feature work is unnamed changesets addressed by id.
• version increments by exactly 1 on every update and starts at 1.

4.3.7 RefLogEntry

(ref, version, old_target, new_target, principal, at, landed: list<(change_id, source, result)>)

The ref log is the system's write-ahead log of main. <ref>@<version> addresses a historical target.

4.3.8 Principal

• github:<login> (humans, via GitHub login), with stable numeric id carried in the token.
• service:<name> (agents, CI, importer), via configured secrets.
• group:<name> (policy only; never authenticates).

4.3.9 Redaction

(object_id, state: Quarantined | Purged, reason, actor, created_at, purge_after, epoch)

4.3.10 Working Copy State

(root_path, endpoint, base: ChangesetId, tracking_ref, sparse: list<path>, inaccessible: list<path>, pending_operation: option<Operation>, index: StatCache)

4.4 Path Rules

• A name is 1-255 bytes, contains no 0x00 and no /, is not . or .., and is not .scion under ASCII case-insensitive comparison.
• Names are arbitrary bytes otherwise (no Unicode normalization; matches git).
• A path is names joined by /, at most 4096 bytes, with no leading or trailing /. The root path is the empty string.
• Paths compare bytewise. Case-only differences are distinct paths; clients on case-insensitive filesystems MUST detect and report collisions during checkout rather than silently overwrite.

4.5 Global Invariants

1. Every stored object's bytes hash to its identifier.
2. A changeset row exists only if its root tree, meta, all parents, all delta blobs, and all eager derived values exist.
3. Nothing is ever mutated in place except refs, counters, policy/redaction state, and lazy derived data write-back (which is idempotent and deterministic).
4. No identifier ever changes, including under redaction.

5. Storage Specification

5.1 Object Storage Layout

• Key: <prefix>/r/<repo_id>/<kind_letter>/<hex[0:2]>/<hex> holding a zstd frame of the content.
• Quarantine key: <prefix>/q/<repo_id>/<kind_letter>/<hex[0:2]>/<hex>.
• Trees and Meta whose encoding is at most objects.inline_threshold_bytes (default 65536) are stored inline in the metadata store instead.

5.2 Metadata Schema

All tables are keyed by repo_id first (single row in repos for v1).

Implementations MAY add secondary indexes. Implementations MUST NOT add columns that participate in object identity.

5.3 Write Ordering

1. Object upload: write bytes to object storage (or prepare inline data), then insert the objects row with ON CONFLICT DO NOTHING. A live row implies durable bytes.
2. Changeset creation: all referenced objects exist and are live (or carried over unchanged from the first parent; Section 13.6) -> write new trees -> compute eager derived values -> one transaction inserting the changesets row, changeset_children, object_paths, and eager derived rows, all with ON CONFLICT DO NOTHING.
3. Ref update: one transaction performing the version-checked update, the ref_log insert, and the landed inserts.

Bytes in object storage without an objects row are orphans. Orphans are harmless and are reclaimed by deferred GC (Appendix A).

5.4 Consistency Rules

• Ref reads at the server MUST be strongly consistent.
• Reads of immutable rows (objects, changesets, derived values) MAY use stale/follower reads; a miss under a stale read MUST be retried with a consistent read before reporting NOT_FOUND.
• Redaction state reads follow Section 13.3 propagation bounds, not stale-read rules.

6. Configuration Specification

All configuration files are TOML. Unknown keys MUST be rejected at startup with a typed error naming the key. String values of the form $NAME resolve from the environment; unresolved variables are startup errors.

6.1 Server Config (scion-server.toml)

[server]
listen = "0.0.0.0:7443"
tls_cert = "/etc/scion/tls.crt"
tls_key = "/etc/scion/tls.key"
repo = "monorepo"

[metadata]
url = "$SCION_DB_URL"
max_connections = 64

[objects]
endpoint = "https://s3.example.internal"
bucket = "scion"
prefix = "prod"
inline_threshold_bytes = 65536
zstd_level = 3

[auth]
github_client_id = "$SCION_GITHUB_CLIENT_ID"
github_api_url = "https://api.github.com"
token_ttl = "12h"
signing_keys = ["/etc/scion/keys/2026-10.ed25519"]   # first key signs; all keys verify

[[auth.service_principals]]
name = "ci"
secret_sha256 = "$SCION_CI_SECRET_SHA256"

[policy]
path = "/etc/scion/acl.toml"
reload = true

[limits]
max_blob_bytes = 104857600
max_delta_entries = 100000
max_parents = 16
max_message_bytes = 1048576
merge_max_text_bytes = 10485760

[land]
max_attempts = 32
backoff_base_ms = 5
backoff_max_ms = 500

[derive]
lease_ttl_ms = 30000
lazy_workers = 16

[redaction]
propagation_bound_ms = 1000
default_quarantine = "7d"

[observability]
log_format = "json"
metrics_listen = "127.0.0.1:9464"

6.2 Access Policy (acl.toml)

version = 1

[groups]
eng = ["github:alice", "github:bob", "group:agents"]
agents = ["service:agent-pool", "service:ci"]
security = ["github:carol"]
admins = ["github:alice"]

[[paths]]
prefix = "/"
read = ["group:eng"]
write = ["group:eng"]

[[paths]]
prefix = "/secrets"
read = ["group:security"]
write = ["group:security"]

[refs."main"]
update = ["group:eng"]
force = []

[refs."release/*"]
update = ["group:admins"]
force = ["group:admins"]

[refs."green/*"]
update = ["service:ci"]
force = ["service:ci"]

[admin]
redact = ["group:admins"]
purge = ["group:admins"]
refs = ["group:admins"]          # create/delete refs

Semantics: