[ { "id": "P00", "title": "Foundation: workspace, frozen API stubs for all crates, scion-core, scion.v1 proto, contract docs, CI scripts", "wave": 0, "spec_sections": "3.2, 3.4, 4.1-4.3 (type shapes), 4.2 (ids and text form), 5.2 (row types), 6.5 (Limits), 7.1-7.8 (complete scion.v1 proto), 7.2 (error table), 8.3 (line-diff3-v1 contract), 9.2 (commit_graph formula), 18.5, 19.0 (fault registry, checkpoint names), 20.1", "owns": [ "Cargo.toml", "Cargo.lock", "rust-toolchain.toml", "rustfmt.toml", "clippy.toml", ".cargo/config.toml", ".gitignore", "crates/*/Cargo.toml (all 23 crates)", "crates/*/src/lib.rs (all 23 crates)", "crates/scion-proto/build.rs", "proto/scion/v1/scion.proto", "crates/scion-core/**", "crates/scion-history/src/{hooks.rs,config.rs,types.rs,graph_compute.rs}", "crates/scion-derived/src/types.rs", "crates/scion-server/src/{grpc.rs,handlers/mod.rs,authz/mod.rs}", "crates/scion-wc/src/points.rs", "initial stub content of every other module file listed in PACKAGES.json (ownership of each transfers to the listed package)", "scripts/{check.sh,check-layers.sh,layers.toml,test-external.sh}", "docker/compose.test.yml", ".github/workflows/ci.yml", "docs/contracts/{line-diff3-v1.md,storage-semantics.md,amendments.md}", "docs/implementation-defined.md", "docs/testing.md", "testdata/README.md" ], "depends_on": [], "deliverables": "Create the whole Cargo workspace so that it compiles with every public API of docs/PLAN.md section 2 present. Concretely: (1) Root Cargo.toml: resolver = 3, edition 2024, an explicit members list of the 23 crates in PLAN 1.2. It has [workspace.package] and [workspace.lints] (rust unsafe_code = deny; clippy unwrap_used/expect_used/panic = deny, with clippy.toml allow-unwrap-in-tests/allow-expect-in-tests = true). It has [workspace.dependencies] pinning EXACT versions, resolved at scaffold time, of every third-party crate in PLAN 5.3, plus the profiles in PLAN 3.5. Generate and commit Cargo.lock. rust-toolchain.toml pins the installed 1.98 toolchain with rustfmt and clippy. (2) Every crate Cargo.toml declares all internal deps from PLAN 1.2 and all third-party deps it will need (PLAN 5.3 column 'used by'), dev-deps included, so later packages never edit manifests. It also declares features: scion-server/testing, scion-daemon/testing, scion-wc/testing, scion-import/testing, scion-auth/test-hooks. Bins: scion-server, scion-daemon, scion (crate scion-cli), scion-import; each src/main.rs is a stub. (3) Fully implement scion-core exactly as PLAN 2.1, with unit tests: error table, ids and text form, empty_tree_id, Mode, conflict types, Principal, Tier, Clock/ManualClock, Limits defaults, FaultRegistry semantics, points, trace, test_support. (4) Write proto/scion/v1/scion.proto completely per PLAN 2.13 (all six services, every message). crates/scion-proto/build.rs uses tonic-prost-build (protoc is on PATH) and emits a file descriptor set. The v1 module and FILE_DESCRIPTOR_SET are real; the other scion-proto modules are stubs. (5) Every other crate gets lib.rs and the module files named in PACKAGES.json owns lists. Each holds every PLAN section 2 item with rustdoc that cites the SPEC section. Bodies follow PLAN 2.0: todo!(\"<owning package id>\") for behaviour; real assembly constructors and accessors (History::new, ObjectRepo::new, Storage::new, Storage::in_memory, MemMetaStore::new, MemBlobStore::new, DerivedEngine::new, StoreDeriveCtx::new, Registry::empty, Redactor::new, ObjectAuthorizer::new, RedactionSet::new); real trivial impls of NoEager and AllowAll; real compute_commit_graph (9.2: generation = 1 + max parent generation, root 1; skip[k] = first-parent ancestor at distance 2^k); no-op handlers::admin::spawn_background. Handler stubs return Err(Status::unimplemented(\"Pxx\")). scion-server/src/grpc.rs is real: it implements the six generated tonic server traits, and each method delegates in one line to the handlers::<file>::<rpc> function listed in PLAN 2.14. (6) docs/contracts/line-diff3-v1.md (normative, PLAN 6.3): line splitting. Myers forward greedy O(ND) with the exact k-selection rule: move down (insert) when k == -d or (k != d and V[k-1] < V[k+1]), else move right (delete). Snake extension, storage of V per d, backtracking with the same rule, no heuristics, no cost cap. Edit-script normalization (within a changed run, deletions are listed before insertions). Hunk extraction. diff3 region building (transitive closure of b->o and b->t hunks whose base ranges overlap or touch, where an empty insertion range touching or inside another range counts as touching). A one-sided region applies. A two-sided region applies once if both sides produce identical text, otherwise it conflicts. Result assembly. At least 12 hand-worked examples, including tie-sensitive ones (swap AB/BA, repeated lines, empty inputs, unterminated last line, CRLF). (7) docs/contracts/storage-semantics.md: for every MetaStore method, the transaction boundary, reads, writes, uniqueness/conflict behaviour, outcome mapping, the CockroachDB DDL/SQL shape (all SPEC 5.2 tables, plus audit_log and secondary indexes: changeset id prefix, derived source_object, redactions epoch, redactions purge_after) and notes on the Mem implementation. Also ObjectRepo write ordering and the key layout. (8) docs/contracts/amendments.md (protocol plus an empty log). docs/implementation-defined.md: transcribe PLAN section 6 verbatim as the documented implementation-defined behaviours. docs/testing.md: summary of PLAN section 3 with env vars and commands. testdata/README.md. (9) Scripts. scripts/check.sh runs fmt check, build, test, clippy -D warnings and check-layers; --full sets SCION_FULL=1 and runs cargo test --release -- --include-ignored. scripts/check-layers.sh plus scripts/layers.toml implement PLAN 1.3 from cargo metadata, with a --self-test mode proving an injected upward edge is rejected. scripts/test-external.sh. docker/compose.test.yml: cockroachdb single-node insecure on 26257 and MinIO on 9000, with bucket creation. .github/workflows/ci.yml: check.sh on push; nightly --full and external jobs. (10) Before finishing, self-review against SPEC.md and PLAN section 2. cargo doc --workspace --no-deps must be warning-free. Gate G0 (a fresh-agent contract review) follows this package. Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "scion-core unit tests: ErrorCode::ALL maps to the exact SPEC 7.2 gRPC status and retryable flag (19.7). ObjectId text form round-trips for every kind; ids with equal digests and different kinds are unequal and print differently (19.1). parse_text rejects bad input. ChangeId hex round-trips. empty_tree_id equals BLAKE3 of [0x01,0x00,0x00] with kind Tree. Principal parse/display. FaultRegistry: arm with times, hits counted when unarmed, Crash latch until reset, Delay. compute_commit_graph on hand-built rows (root generation 1; skip pointers at distances 1, 2, 4, 8 on a 10-chain; merge generation = 1 + max). Limits::default matches SPEC 6.5. scripts/check-layers.sh passes, and --self-test proves an upward edge is rejected. cargo build, clippy and doc are clean.", "size": "XL" }, { "id": "P01", "title": "scion-object: canonical codec, entities, path/ref/revision grammar, golden vectors", "wave": 1, "spec_sections": "4.1, 4.2 (ids inside objects), 4.3.1-4.3.4, 4.3.6 (ref name grammar), 4.4, 9.2 (HistoryNode encoding, derived kind 0x10), 11.8 (revision grammar parsing), 17 (decoded-form validation)", "owns": [ "crates/scion-object/src/{enc.rs,canonical.rs,extensions.rs,path.rs,tree.rs,meta.rs,changeset.rs,history_node.rs,refname.rs,revision.rs,flat.rs}", "crates/scion-object/tests/**", "testdata/golden/objects/**" ], "depends_on": [ "P00" ], "deliverables": "Implement PLAN 2.2 completely. Encoder/Decoder: uvarint is minimal LEB128 of at most 10 bytes, with overflow rejected; svarint is zigzag; bytes/list lengths may not exceed the remaining input; option tag must be 0 or 1; finish() rejects trailing bytes. All errors are INVALID_ARGUMENT with the scion-core decoding subcodes. Extensions: strictly ascending unique tags. Unknown odd tags are rejected (unknown_critical_extension). Even tags are preserved byte-for-byte; v1 has no known odd tags. Tree: format_version must be 1 (else unknown_format_version). Entries are strictly ascending by bytewise name and unique; names obey 4.4. Modes 1-3 require a Blob id, mode 4 requires a Tree id with size 0. validate_non_root rejects an empty tree (empty_tree). Meta: fields in SPEC order; Timestamp is a pair of svarints. Changeset: parents of kind Changeset, unique; root_tree of kind Tree; meta of kind Meta; change_id is 16 raw bytes; delta strictly ascending by path, op 1 = Put (mode 1..=3, id of kind Blob, size) and 2 = Delete. validate_shape(limits) checks parent count/uniqueness, delta sort/uniqueness, max_delta_entries and path rules (TOO_LARGE for limits, with a Limit payload). HistoryNode: format_version 1, kind u8 (1 File, 2 Dir), content, changeset, parents list, children list<(name, id)> ascending (must be empty for File), extensions. KIND = Derived(0x10). Name/RepoPath: 1-255 bytes, no NUL or '/', not '.' or '..', not '.scion' under ASCII case-insensitive comparison; paths are at most 4096 bytes with no leading/trailing '/' and no empty component; the root is the empty path; ancestors() and is_ancestor_of are component-wise. RefName: ^[a-z0-9][a-z0-9._/-]{0,199}$, with no '..', no '//' and no trailing '/'. parse_revision covers: full id text; 'c1' plus 8..63 hex (IdPrefix); a bare name, treated as a ref name or a bare hex prefix of at least 8 digits, decided later by the server; '<ref>@<u64>'; 'git:<40 hex>'; and chains of '~n' / '^k' suffixes (a bare '~' or '^' means 1; ^0 is invalid). flat::file_diff implements SPEC 4.3.4 FileDiff over FlatTree. Golden vectors: testdata/golden/objects/*.json with at least 14 cases: empty tree; a tree with all four modes; a tree with an even extension; a Meta with and without committer; a Meta with extension tag 2; the root changeset; a changeset with 2 parents and mixed Put/Delete; a changeset with an even extension; HistoryNode File and Dir; a blob. Each file is {name, kind, description, hex, id_text}. A test re-encodes every case from its constructor in Rust and compares bytes and id. A second test asserts a BLAKE3 digest constant over the sorted golden file contents, so any edit to the files fails CI (19.1 freeze). Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "19.1: golden vectors frozen (encode equality plus digest constant). Round trip decode(encode(x)) == x and encode(decode(b)) == b with proptest over random valid Tree/Meta/Changeset/HistoryNode. The default count is proptest_cases(10_000); a #[ignore] full_ test runs 10^6. One rejection test per class: non-minimal varint, 11-byte varint, length overrun, trailing bytes, unsorted entries, duplicate entries, unknown format_version, unknown odd extension tag, invalid mode, mode/kind mismatch. An unknown even tag is preserved byte-for-byte and changes the id. Equal digests with different kinds are never confused (a Tree id is rejected where a Blob id is required). 19.2 path-rule violations at parse level: '.', '..', '/', NUL, '.scion'/'.SCION'/'.Scion', 256-byte name, 4097-byte path, leading/trailing '/'. Ref-name grammar table. Revision grammar parse table (at least 25 cases, valid and invalid). file_diff unit cases (file<->dir transitions produce Delete plus Put).", "size": "M" }, { "id": "P02", "title": "scion-merge: normative Myers diff and line-diff3-v1 engine", "wave": 1, "spec_sections": "8.3 (content merge, binary detection, decision table), 9.2 (blame line mapping), 7.4 (Diff hunks), 11.7 (marker format), 19.4", "owns": [ "crates/scion-merge/src/{split.rs,myers.rs,hunks.rs,merge3.rs,binary.rs,classify.rs,markers.rs,origins.rs,unified.rs}", "crates/scion-merge/tests/**", "testdata/golden/diff/**" ], "depends_on": [ "P00" ], "deliverables": "Implement PLAN 2.5 exactly according to docs/contracts/line-diff3-v1.md. That document is normative; if it is ambiguous, follow its worked examples and file an amendment. Do not use any third-party diff crate in src (similar is allowed only as a dev-dependency to cross-check edit-script minimality). split_lines keeps terminators. myers is the forward greedy O(ND) algorithm with the contract's k-selection rule and backtracking. diff_lines groups non-equal runs into hunks. is_binary: a NUL within the first 8000 bytes. merge3: region building, one-sided apply, identical two-sided apply-once, conflicts reported as ConflictHunk line ranges (base/ours/theirs, u32), and byte-exact output (no CRLF normalization, unterminated final lines preserved). classify implements 8.3 rows 1-6 in order: o==b take t; t==b take o; o==t take o; b absent with o and t present and different gives AddAdd; exactly one side absent and the other differing from b gives ModifyDelete; any side a symlink with sides differing gives Symlink; otherwise NeedContent. merge_mode applies the first three rows to modes. render_markers writes the exact 11.7 marker lines. line_origins maps child lines to parent lines via diff_lines (for blame). unified_hunks produces unified-diff hunks with context. Produce at least 60 golden vectors in testdata/golden/diff/*.json: diff cases {a_hex, b_hex, edits} and merge cases {base_hex, ours_hex, theirs_hex, result_hex | conflicts}. Include all worked examples from the contract doc and tie-break-sensitive inputs. Freeze them with a digest-constant test. Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "19.4 content level: o==b => t; t==b => o; o==t => o; non-overlapping, non-adjacent hunks from both sides both apply; overlapping hunks conflict; adjacent hunks (zero unchanged lines between) conflict; identical hunks apply once. Each property runs with proptest_cases(10_000), with #[ignore] full_ variants at 10^6. Binary boundary: NUL at byte 7999 is binary, at byte 8000 is text. CRLF and unterminated-final-line merges are byte-exact. classify has one test per 8.3 table row. Myers optimality: edit length equals an O(N*M) LCS-based minimum on small random inputs. Determinism: the same input always gives the same output. Golden vectors are frozen. RECOMMENDED: a differential against `git merge-file -p` on clean text merges (skipped when git is absent), with disagreements recorded in crates/scion-merge/tests/GIT_MERGE_FILE_DIFFERENCES.md.", "size": "M" }, { "id": "P03", "title": "scion-store: in-memory MetaStore/BlobStore, ObjectRepo, counters and fault decorators, fsck, backend conformance suites", "wave": 1, "spec_sections": "5.1, 5.2, 5.3 step 1, 5.4, 4.5 (fsck), 9.1 (leases), 13.2 (redaction transactions and byte moves), 16.1 classes 2/3/7, 17 (audit log, decoded-form validation), 19.0 (counters, fault hooks), 19.3", "owns": [ "crates/scion-store/src/{types.rs,meta.rs,blob.rs,storage.rs,keys.rs,mem_meta.rs,mem_blob.rs,repo.rs,instrument.rs,fsck.rs,conformance.rs}", "crates/scion-store/tests/**" ], "depends_on": [ "P00" ], "deliverables": "Implement PLAN 2.3 and docs/contracts/storage-semantics.md. MemMetaStore: all state is im persistent maps behind one parking_lot::Mutex, and each trait method is one atomic step (strictly serializable). Required behaviour: - ON CONFLICT DO NOTHING for objects, changesets, children, object_paths and derived; commit_changeset inserts every row independently, so a re-commit fills in missing rows. - cas_ref: RefExpect::Absent or Version(v); on success version becomes v+1 (1 on create); ref_log row, landed rows and audit row in the same step; a landed (ref, change_id) unique violation returns LandedConflict and applies nothing; a version mismatch returns the current row. - redact: increments counters.redaction_epoch, inserts rows stamped with the new epoch, sets objects.state = quarantined. unredact: Quarantined only (else INVALID_ARGUMENT/not_quarantined), epoch+1, state live. mark_purged: clears inline_data, state purged. - Leases are TTL-based against the provided now. - A ring of the last K committed snapshots serves Consistency::Stale (stale_lag_commits, set_stale_lag, advance_stale_view). - fail_txn_every: every Nth transaction fails with UNAVAILABLE/txn_retry and no effect. MemBlobStore supports corrupt(), set_down() and keys(). keys.rs implements the SPEC 5.1 layout. InstrumentedMeta/InstrumentedBlob count meta_txns, meta_rows_read, object_paths_rows_read, lease_calls and blob ops/bytes, and check fault points meta.<method>.before / meta.<method>.after and blob.<method>.before / blob.<method>.after (FailAfter means the effect is applied and the call then fails). Leases emit a tracing span with target scion::lock. ObjectRepo: - put_blob/put_encoded/put_verified validate size (TOO_LARGE) and the decoded form of known kinds (Tree/Meta/Changeset/HistoryNode), and reject redacted ids (REDACTED). - Blobs always go to object storage. Tree/Meta/Derived objects at or below inline_threshold_bytes are stored inline, others as zstd frames at zstd_level. - Write order: bytes first (checkpoint upload.after_bytes), then the objects row (checkpoint upload.after_row); a live row therefore always implies durable bytes. - get/get_frame verify BLAKE3 of the decompressed content; on mismatch they record an integrity failure (Counters plus an ERROR log event 'integrity_alert') and return INTEGRITY. - Every read records object reads by kind and bytes. - stat/get use a stale read first and retry a miss with a strong read before NOT_FOUND. - LRU caches of decoded trees, metas and changesets serve immutable reads even while the metadata store is down. - quarantine_bytes copies to the quarantine key and then deletes the live key (checkpoint quarantine.after_copy). restore_bytes reverses it. purge_bytes deletes both keys. fsck walks every changeset and checks SPEC 4.5: digest of every live object, presence of root tree, meta, parents and delta blobs, delta == FileDiff(parent tree, root tree), generation and skip consistency, required eager rows, and ref targets exist. Quarantined or purged bytes are not violations. conformance::meta_store_suite and blob_store_suite are public async fns (always compiled, a test-support API). They cover every method contract, including the concurrency cases below, and run against MemMetaStore/MemBlobStore in this package's tests. P15 reuses them for CRDB/S3. Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "Tests in crates/scion-store/tests: - meta_store_suite and blob_store_suite on Mem. - CAS: 1000 concurrent cas_ref calls at the same expected version give exactly one Committed; versions advance by exactly 1; ref_log has no gaps (19.5, store level). ABA: A->B->A, then a CAS expecting the first version fails. - LandedConflict rolls back the whole cas_ref. - 19.3: a blob-store fault before put leaves no objects row; FailAfter on put still leaves no row unless the row insert ran; a row never exists without bytes. Inline threshold at threshold-1, threshold and threshold+1 stores in the expected place and reads back identically. A stale-read miss is retried strongly (stale_lag) before NOT_FOUND. Orphan bytes (a put with no row) are invisible to stat/get. - 19.1: a blob id is independent of zstd level (1, 3, 19) and framing (a multi-frame encoding of the same content). - Corrupt stored bytes give INTEGRITY and integrity_failures +1 (16.1 class 7). - fail_txn_every leaves no partial state. - Redaction epoch increments atomically with the state changes. - Counters count exactly (tree reads include inline trees). - fsck detects each Violation kind on hand-corrupted state and passes on valid state.", "size": "L" }, { "id": "P04", "title": "scion-config and scion-obs: TOML configs, structured logging, metrics, health", "wave": 1, "spec_sections": "6.1, 6.3, 6.4, 6.5, 15.1, 15.2, 15.3, 16.1 class 7 (integrity alert metric), 19.15, 19.17", "owns": [ "crates/scion-config/src/**", "crates/scion-config/tests/**", "crates/scion-obs/src/**", "crates/scion-obs/tests/**" ], "depends_on": [ "P00" ], "deliverables": "Implement PLAN 2.11 and 2.12; lib.rs and Cargo.toml stay P00's. Config: - serde with #[serde(deny_unknown_fields)] everywhere, wrapped in serde_path_to_error so ConfigError::UnknownKey names the full dotted key (for example 'limits.max_blobs'). - '$NAME' string values are resolved from Env in a pre-pass over toml::Value, keeping the key path. An unresolved variable is ConfigError::UnresolvedVar naming the variable and the key. - Every default from SPEC 6.5 and PLAN 2.11 (including the implementation-defined (+) keys) applies when a key is omitted. - Durations use humantime ('12h', '7d'); millisecond keys are integers. - resolve_endpoint precedence: flag > SCION_ENDPOINT > working-copy config > user config. - for_tests constructors. Obs: - init_logging emits JSON (one event per line) or text. - RpcLogEvent::emit logs the REQUIRED 15.1 fields: request_id, rpc, tier, principal, repo, status, duration_ms, and when applicable changeset_id, ref, ref_version, land_attempt. - path_hash returns 16 hex characters of BLAKE3. Raw repository paths are never logged (PLAN 6.9 log access policy). - Metrics registers every SPEC 15.2 metric with the exact names and labels (scion_rpc_duration_seconds{rpc,tier,code}, scion_cache_requests_total{tier,kind,result}, scion_cache_bytes{tier}, scion_land_attempts{result}, scion_ref_cas_failures_total{ref}, scion_land_duration_seconds, scion_derive_duration_seconds{type,mode}, scion_derive_backlog{type}, scion_redaction_epoch{tier}, scion_redaction_lag_seconds{tier}, scion_policy_epoch_info{epoch}, scion_object_bytes_served_total{tier,encoding}, scion_upstream_requests_total{tier}), plus scion_integrity_failures_total{tier}. NAMES lists them all. - health_router serves /metrics (text exposition), /healthz (200 while alive) and /readyz (200 or 503 from Readiness::check). serve_admin binds it. - capture::LogCapture installs a process-global in-memory JSON layer for tests, with find_needles for log scans. Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "19.15: an unknown key is rejected with the key name, at top level and in a nested section, for server, daemon and client configs. $VAR resolves. An unresolved $VAR errors, naming the variable and key. Every SPEC 6.5 default is applied when its key is omitted (one assertion per row of the 6.5 table, plus the (+) keys). The SPEC 6.1 example file parses with env vars supplied. 19.17 unit level: every 15.2 metric name appears in render() after one observation; /healthz returns 200; /readyz returns 503 then 200 as a toggled Readiness changes; RpcLogEvent produces all required fields as JSON keys; path_hash is stable and never equals the input; LogCapture.find_needles finds a planted secret.", "size": "M" }, { "id": "P05", "title": "scion-policy: acl.toml parsing, evaluation, epoch, hot reload", "wave": 1, "spec_sections": "6.2, 12.4, 4.3.8, 16.1 class 9, 19.12 (policy semantics and reload)", "owns": [ "crates/scion-policy/src/**", "crates/scion-policy/tests/**" ], "depends_on": [ "P00" ], "deliverables": "Implement PLAN 2.9. Parse with toml and serde (deny_unknown_fields; UnknownKey names the key). version must be 1. A '/' rule is REQUIRED (MissingRootRule). Groups: nested groups are resolved; cycles are a GroupCycle error listing the cycle; unknown groups are rejected. Principal forms: 'github:<login>' (compared case-insensitively), 'github-id:<n>' (matches Principal::GitHub{id: Some(n)}), 'service:<name>', 'group:<name>'. Prefixes are written with a leading '/' and normalized (no trailing '/'). Matching is component-wise, so '/secrets' matches secrets and secrets/x but not secrets2. For each path, the longest matching rule governs, independently for read and for write; rules never merge. read_governed_by_root reports whether the governing read rule is '/'. Refs: an exact name beats any pattern; among patterns (a single trailing '*' segment) the longest wins. Admin permissions: redact, purge, refs. Default deny. epoch = first 8 bytes, big-endian, of BLAKE3 over a canonical normalized serialization (sorted groups, rules and principals), so that equivalent files agree. PolicyHandle: arc-swap of Arc<Policy>; load() fails on an invalid or missing file. reload_now swaps atomically only when the new policy is valid. An invalid file keeps the last good policy, records last_error and logs an operator-visible ERROR. A watch channel publishes epoch changes. spawn_watcher uses notify plus a 1 s polling fallback. Tests use reload_now for determinism. allow_all_for_tests() grants everything to everyone. impl From<PolicyError> for ScionError (INVALID_ARGUMENT). Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "19.12 policy semantics: longest prefix wins; component-wise matching ('/secrets' vs '/secrets2'); independent read and write resolution; nested groups; cycle detection; the github-id: form; case-insensitive github logins; ref pattern precedence (exact > longest pattern) and default deny; admin permissions. Reload: a valid change is applied atomically and the epoch changes; an equivalent reformatted file gives the same epoch; an invalid change keeps the last good policy, surfaces last_error and leaves the epoch unchanged; an invalid or missing file at startup fails. A concurrency test: readers never observe a torn policy during repeated reloads.", "size": "M" }, { "id": "P06", "title": "scion-auth: Ed25519 PASETO tokens, keyring rotation, constant-time secrets, GitHub client, device flow, FakeGitHub", "wave": 1, "spec_sections": "12.1, 12.2, 12.3, 17 (secrets never logged), 16.1 class 10, 19.11", "owns": [ "crates/scion-auth/src/**", "crates/scion-auth/tests/**", "crates/scion-testkit/src/fake_github.rs", "crates/scion-testkit/tests/fake_github_*.rs" ], "depends_on": [ "P00" ], "deliverables": "Implement PLAN 2.10 and the FakeGitHub part of PLAN 2.21. Tokens: PASETO v4.public via pasetors (Ed25519). Claims sub, gid, iat, exp. kid goes in the footer as JSON {\"kid\":...}; kid = first 16 hex characters of BLAKE3(public key). Keyring::from_files reads PKCS#8 PEM Ed25519 keys. The first key signs and all keys verify. generate_for_tests and write_pem_files support harnesses. Verifier::verify returns UNAUTHENTICATED for malformed, unknown-kid, bad-signature, tampered and expired tokens (exp <= now). ct_eq wraps subtle::ConstantTimeEq and is the ONLY comparison primitive for secrets; under feature test-hooks it counts calls. ServicePrincipals::verify always computes SHA-256(secret) and calls ct_eq, comparing against a dummy digest for unknown names. Secret has redacting Debug/Display. GitHubClient::user calls GET {api_url}/user with 'Authorization: Bearer'. 401/403 maps to UNAUTHENTICATED; transport errors and 5xx map to UNAVAILABLE. device_flow::start/poll implement the GitHub device flow (POST {oauth_base}/login/device/code, POST {oauth_base}/login/oauth/access_token), handling authorization_pending, slow_down and expired_token. FakeGitHub (axum on 127.0.0.1:0) serves GET /user (tokens registered with add_user), the device-code endpoints (approve_device_code completes the flow) and a set_down switch returning 503. Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "19.11 token items: a valid token verifies with the right sub and gid; expired, malformed, wrong-key, tampered-payload and tampered-footer tokens are rejected. Rotation: a token signed by key 2 verifies while key 2 is listed and fails once it is removed; new tokens are signed by the first key. Service secret: right succeeds, wrong fails, unknown name fails; ct_eq_calls increases on BOTH the right and the wrong path (the constant-time primitive is used on both). GitHubClient against FakeGitHub: valid token maps login and id; invalid token gives UNAUTHENTICATED; FakeGitHub down gives UNAVAILABLE. A device-flow round trip against FakeGitHub, including authorization_pending. Secret Debug output never contains the secret.", "size": "M" }, { "id": "P07", "title": "scion-refmodel: independent reference model (19.0 oracle)", "wave": 1, "spec_sections": "4.1-4.4, 4.3.4 validity, 8.1 (validation outcomes), 8.2, 8.3, 8.4 (sequential land semantics), 9.2 (commit_graph, history TREESAME, blame), 11.2-11.3 (materialization, sparse scope), 13.2 (--all-versions), 13.5 (redacted content as empty)", "owns": [ "crates/scion-refmodel/src/**", "crates/scion-refmodel/tests/**" ], "depends_on": [ "P00" ], "deliverables": "Implement PLAN 2.20 as a deliberately naive, single-threaded oracle. INDEPENDENCE RULES: the only third-party dependency is blake3 (and std). Do NOT read or copy any production crate under crates/ other than this one; work only from SPEC.md and docs/contracts/line-diff3-v1.md. check-layers.sh verifies that there are no scion dependencies. Contents: - Its own canonical encoder (enc) for trees (built recursively from the flat MTree), Meta and Changeset, plus id text. - apply_delta, which flattens, applies and rebuilds, returning the 8.1/18.2 subcode strings on invalid deltas (delete_absent, delete_not_file, noop_put, file_directory). - file_diff. - Its own Myers and diff3 written from the contract doc, using the simplest correct code, with no optimizations. - merge_trees implementing the full 8.3 table, both structural file_directory rules, the binary and merge_max_text_bytes rules, and redacted conflicts. - MRepo: commit graph (generation, skip pointers by naive walk), is_ancestor (full BFS), merge_bases (full ancestor-set intersection minus dominated), rebase per 8.2 (fast path, EMPTY), log_path with TREESAME semantics equivalent to the 9.2 history node-reuse rule (follow parents[0] when the content equals parents[0]'s, otherwise all parents where the path exists), blame per 9.2 (first parent, then subsequent parents, remaining lines to the commit; redacted content empty), all_versions (distinct contents ever held). - land_sequential. - materialize(sparse) per 11.2. O(n^2) algorithms are acceptable. Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "Self-tests on hand examples taken from the SPEC tables: every 8.3 row and both structural rules; the 18.2 apply_delta subcodes; root changeset rules; generation and skip on a hand DAG including an octopus merge; a criss-cross DAG giving two merge bases; the TREESAME log on a merge that keeps the first parent's version; blame on a 3-commit example; all worked examples of docs/contracts/line-diff3-v1.md reproduced exactly; materialize with nested sparse prefixes. Property: file_diff(t, apply_delta(t, d)) == d for random t and random valid d (proptest_cases(5_000)). Cross-checks against production code are done later by P17 and other packages.", "size": "L" }, { "id": "P08", "title": "scion-proto: conversions, error mapping, epoch trailer layer, ScionClient, RawCodec; testkit FakeUpstream and TestTls", "wave": 1, "spec_sections": "7.1, 7.2, 7.3-7.7 (message mapping), 7.8 (verbatim forwarding codec), 3.3, 17 (TLS), 19.7 (error mapping, forward compatibility)", "owns": [ "crates/scion-proto/src/{convert.rs,status.rs,trailers.rs,client.rs,endpoint.rs,raw.rs}", "crates/scion-proto/tests/**", "crates/scion-testkit/src/{fake_upstream.rs,tls.rs}", "crates/scion-testkit/tests/upstream_*.rs" ], "depends_on": [ "P00" ], "deliverables": "Implement the Rust surface of PLAN 2.13 (the proto file and codegen are P00's and frozen), plus FakeUpstream and TestTls from PLAN 2.21. convert: From/TryFrom between v1 messages and scion-core/scion-object/scion-store types (ObjectId bytes, EntryInfo, DeltaEntry, MetaFields, ConflictReport, LandedEntry, RefRow/RefLogRow, Limits, ServerInfo pieces). A missing required field or a malformed id gives INVALID_ARGUMENT/missing_field or bad_id_text. Unknown fields are ignored (prost default). status: status_from_error sets the tonic code from ErrorCode::grpc_code and the message, and puts the ErrorDetail encoding (code string, retryable, message, payload oneof) into Status details. error_from_status inverts it; a Status without details maps from its gRPC code, defaulting to UNAVAILABLE. trailers: TRAILER_* constants, read_epochs/write_epochs. EpochTrailerLayer is a tower Layer over http Request/Response bodies. It appends scion-policy-epoch, scion-redaction-epoch and scion-tier to the HTTP/2 trailers frame of EVERY response: unary, server-streaming, client-streaming, error, and trailers-only responses (where tonic puts grpc-status in the headers, add the three keys to those headers). Epoch values are decimal u64. endpoint: Endpoint::parse accepts 'unix:///abs/path' and 'https://host:port'. client: ScionClient::connect uses tonic Channel over a UnixStream connector (hyper-util) or https with rustls (CA from TlsOptions, otherwise native roots). It installs a bearer-token interceptor and a response layer capturing the last epochs from trailers (and trailers-only headers), and keeps an atomic call counter (calls/reset_calls). raw: RawCodec is a tonic Codec whose Encode and Decode are both bytes::Bytes, for byte-level forwarding by the daemon. TestTls::generate creates an rcgen CA plus a server certificate for localhost and 127.0.0.1. FakeUpstream: an in-memory tonic server on 127.0.0.1:0 (TLS optional, via TestTls) implementing: ObjectService (GetServerInfo with configured public keys; GetObjects with per-item ok/not_found/denied/redacted, zstd or raw frames, offset/length ranges, access_class, kill_stream_after_bytes; HasObjects; ResolveRef); QueryService GetChangesets and Resolve; AuthService CheckAccess and WhoAmI; AdminService ListRedactions. Every other method returns UNIMPLEMENTED. The principal comes from the bearer token via the configured principal_of closure. The server is wrapped in EpochTrailerLayer using set_epochs values (tier server). It has set_down, corrupt, redact and deny knobs, and records every request (method, ids, principal). scion-proto tests must not dev-depend on scion-testkit (dependency rule PLAN 1.2); proto-level transport tests use rcgen directly. Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "19.7: every ErrorCode round-trips through status_from_error/error_from_status with the correct tonic code and retryable flag, and every payload variant round-trips. A request with unknown fields (hand-encoded extra tag) decodes. Missing required fields in CreateChangesetRequest/MergeRequest give INVALID_ARGUMENT/missing_field. Property round trip for conversions. Transport tests (crates/scion-proto/tests, rcgen certificates, a minimal ObjectService implementation defined in the test): over TLS and over a Unix socket, ScionClient sees all three trailers on a unary success, a unary error, a server-streaming response, a client-streaming response and a trailers-only error, and last_epochs is updated; the call counter counts. RawCodec forwards a unary and a streaming call byte-identically through a test proxy. FakeUpstream tests (crates/scion-testkit/tests/upstream_*.rs): batch mixing ok/not_found/denied/redacted; range read; stream kill and resume by offset; requests are recorded.", "size": "L" }, { "id": "P09", "title": "scion-testkit corpus: generators, shared-subtree repos, edge cases, vendored git bundles, linearizability checker", "wave": 1, "spec_sections": "19.0 (Corpus, History Recorder), 19.5 (linearizability checker), 19.14 (real git histories), 19.18 (equal-depth generated repos)", "owns": [ "crates/scion-testkit/src/{corpus.rs,shared.rs,edge.rs,encode.rs,git.rs,lincheck.rs}", "crates/scion-testkit/tests/corpus_*.rs", "crates/scion-testkit/tests/lincheck_*.rs", "corpus/git/**" ], "depends_on": [ "P00" ], "deliverables": "Implement PLAN 2.21 except fake_upstream.rs and tls.rs (P08) and fake_github.rs (P06). corpus::generate(CorpusSpec): a seeded (rand_chacha) generator of GenRepo histories of structural MTree snapshots, with configurable file count, depth, fanout, history length, merge ratio, octopus ratio, binary ratio and symlink ratio, plus criss-cross shapes. Text edits are line-level, so merges are meaningful; output is deterministic per seed. shared::shared_subtree_repo(files, depth): a balanced tree of identical subtrees reaching 10^3, 10^5 or 10^7 logical files at the SAME depth (fanout chosen per size, depth fixed), perturbable along one path. encoded_objects() returns only the O(depth x fanout) distinct objects. edge::edge_cases(): empty root, deep path at the 4096-byte limit, 255-byte names, case-only-differing names, file<->dir flips, CRLF, NUL content (binary), executable flips, symlink retargets, delete-and-readd, unterminated last lines. encode::encode_tree uses scion-object to produce production-encoded trees and blobs from an MTree. It is exercised first by P17; this package must not run scion-object code in its own tests, because P01 is in the same wave. git: vendor at least two REAL open-source git histories as bundles under corpus/git/ (candidates: rust-lang/cfg-if and dtolnay/itoa, both MIT OR Apache-2.0 and small with merge commits). Verify license and size (each at most 5 MB, full history via `git bundle create <f> --all`), and add corpus/git/README.md plus LICENSE-<repo> files recording source URL, commit and license. Also build and commit synthetic edge bundles with the git CLI: an octopus merge, multiple roots, empty commits, mode 100664, symlinks, a gitlink, a merge keeping the first parent's version, and rename-free line edits for blame. Tests must then not require the git CLI; bundles(), unpack() (via gix) and GitFixture (which needs git and skips when it is absent) are provided. If network access is impossible, stop and report instead of substituting synthetic history for the real-repository requirement. lincheck: HistoryRecorder (thread-safe, monotonic timestamps) and check_ref_registers, a WGL linearizability search with memoization over a versioned-register model per ref. Read returns (version, target); Cas succeeds iff its expect equals the current version; Land succeeds with version+1. An Unknown completion may or may not have taken effect (crash or timeout). Histories are partitioned per ref; document the cap of about 200 operations per history. Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "Generators are deterministic per seed and realize the requested shapes (octopus, multiple roots, criss-cross detected structurally). The shared-subtree repo has the same depth at 10^3, 10^5 and 10^7 files, and its distinct structural subtree count is O(depth x fanout) (computed structurally). Edge-case catalog sanity. lincheck accepts known-linearizable histories (sequential; concurrent with overlap; with Unknown completions) and rejects known violations (stale read after a completed CAS, two successful CAS at the same version, a version gap) on hand-built cases. Bundles exist, their README and licenses are present, and the size limits hold (no git CLI required).", "size": "L" }, { "id": "P10", "title": "scion-cache: verified disk cache, coalescer, upstream limiter, ref/redaction/access caches", "wave": 1, "spec_sections": "10.2, 10.3, 10.4 (decision caches), 13.3 (daemon redaction set and staleness), 16.1 classes 6/7, 19.9", "owns": [ "crates/scion-cache/src/**", "crates/scion-cache/tests/**" ], "depends_on": [ "P00" ], "deliverables": "Implement PLAN 2.15. DiskCache layout: <dir>/objects/<kind_letter>/<hex[0:2]>/<hex>, holding the payload as received (a zstd frame or raw bytes, with a 1-byte header distinguishing them). Insert writes a temp file in the same directory, verifies BLAKE3 of the decompressed content against the id (INTEGRITY, nothing cached, on mismatch), fsyncs and renames. open() deletes leftover temp files and rebuilds the size and access index by scanning. get verifies on read when verify_on_read is set; a mismatch deletes the entry and returns Ok(None) so the caller refetches. Approximate LRU by last access time: after an insert that pushes usage over max_bytes, evict down to low_watermark x max_bytes. usage() is exact. Coalescer is single-flight per key: concurrent callers share one future, errors are shared, and the entry is removed after completion. UpstreamLimiter: semaphore(max_concurrency) plus a bounded waiter count; overflow gives RESOURCE_EXHAUSTED immediately. RefCache: TTL-bound, and never returns a lower version than previously returned for a ref, even after TTL expiry or out-of-order upstream answers. LocalRedactions: id set, epoch, last refresh and is_stale. AccessCache: entries keyed by (principal, object) with TTL, plus a per-principal root-reader flag with TTL. Decisions never leak across principals, and on_policy_epoch(epoch) clears everything when the epoch differs. Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "19.9 unit level: a corrupt insert is rejected and not cached; a bit flip in a cached file is detected on read (verify_on_read), deleted and get returns None; temp files are removed on open; a simulated crash after the temp write and before rename leaves no visible entry; 10000 concurrent get_or_fetch calls for one key run the fetch exactly once; a randomized insert/get workload keeps usage <= max_bytes at all times and evicted ids read back as None; ref TTL is honored and versions stay monotonic across TTL expiry and reordered observations; queue overflow gives RESOURCE_EXHAUSTED. AccessCache isolates principals and clears on an epoch change.", "size": "M" }, { "id": "P11", "title": "scion-wc local layer: safe filesystem, stat-cache index, status engine, ignore, state, lock, credentials", "wave": 1, "spec_sections": "4.4 (path rules, case collisions), 6.4 (credentials), 11.1, 11.4, 11.5, 17 (checkout path safety, credentials 0600), 18.7, 19.10 (status, ignore, path safety), 19.11 (credentials)", "owns": [ "crates/scion-wc/src/{fs_safe.rs,index.rs,status.rs,ignore.rs,state.rs,lock.rs,credentials.rs}", "crates/scion-wc/tests/local_*.rs" ], "depends_on": [ "P00" ], "deliverables": "Implement the corresponding modules of PLAN 2.17. SafeRoot uses rustix openat with O_NOFOLLOW for every path component (a pre-existing symlinked parent is refused, never followed). Writes go to a temp file plus renameat in the target directory, with the exec bit set for Mode::Executable; symlink() creates symlinks; nothing is ever written outside the root. is_case_insensitive probes the filesystem once. validate_entry_path enforces 4.4 and rejects '.scion' in any case. case_collisions uses ASCII case folding, so it is deterministic on any filesystem; checkout uses it whenever the filesystem is case-insensitive or WcOptions.force_case_insensitive is set. StatCache: (size, mtime_ns, ctime_ns, inode, mode) tuples plus written_at_ns, stored as a binary file .scion/index (versioned header) and saved atomically. is_racy means mtime >= written_at. status follows 18.7 over the TreeLookup trait: walk in-scope files, skip .scion and ignored files, skip inaccessible prefixes; hash files (symlinks hash their target) only when the stat tuple changed or the entry is racy; report modified/added/deleted/mode_changed; refresh clean entries; an unmodified redacted placeholder (index redacted flag plus placeholder bytes) is clean, and an edited placeholder is reported in redacted_dirty and modified; count hashed files. IgnoreMatcher uses ignore::gitignore (nested .gitignore files, negation, directory patterns, anchoring) with git semantics. state: WcState and PendingOperation as canonical TOML (sorted keys, deterministic), saved atomically under .scion/state. lock: fs4 advisory non-blocking lock on .scion/lock. credentials: write creates the file with mode 0600 (and its parent with 0700); read refuses a group- or world-readable file with PERMISSION_DENIED and a clear message. Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "19.10 status: randomized edit sequences (create, modify, delete, chmod +x/-x, symlink swap, rename, truncate-and-write, atomic-rename save, edits within the same second as the index write, i.e. racy-clean) match a full-rehash oracle after every step (proptest_cases(500); #[ignore] full_ at 10^4). Clean status hashes only racy or stat-changed files. .gitignore semantics match `git check-ignore` on a corpus of at least 40 patterns (skipped when git is absent), and a pure table of expected results runs always. Path safety: '..', '.scion', '.SCION' and NUL paths are refused; writes through a pre-existing symlinked parent directory are refused and nothing is created outside the root. case_collisions reports collisions (deterministic on any fs; plus a real collision test when the tmp fs is case-insensitive, as on macOS APFS). State and index round-trip and are atomic (a crash simulated mid-save leaves the old file). The lock excludes a second holder. 19.11: the credentials file is created 0600; read refuses 0640 and 0604.", "size": "L" }, { "id": "P12", "title": "scion-history: tree reading, apply_delta, FileDiff, CreateChangeset, bootstrap", "wave": 2, "spec_sections": "4.3.2, 4.3.4, 4.5, 5.3 step 2, 8.1, 9.2 (commit_graph columns, path_index rows), 13.6, 14 (parentless import_mode), 18.1, 18.2, 16.1 class 4, 19.2, 19.3, 19.18 (CreateChangeset gate)", "owns": [ "crates/scion-history/src/{history.rs,tree.rs,apply.rs,diff.rs,create.rs,bootstrap.rs}", "crates/scion-history/tests/create_*.rs" ], "depends_on": [ "P01", "P03", "P07" ], "deliverables": "Implement the P12 part of PLAN 2.6 (History::new and accessors already exist). tree/lookup/changeset go through ObjectRepo caches. apply_delta follows 18.2 exactly: walk only the ancestors of delta paths, deepest first, applying deletes, then rebuilt children, then puts. Subcodes: delete_absent, delete_not_file, file_directory (ancestor file not deleted, or a directory not fully deleted), noop_put. An empty non-root directory is pruned; the root may become empty. Return the new root plus the new non-empty trees with their canonical bytes. file_diff prunes identical subtree ids and supports a scope filter. create_changeset validates in the exact 8.1 order with the first failure returned: - parents exist, are unique and within max_parents (TOO_LARGE); empty parents only when has_root_changeset() is false, or when import_mode is set and authz.can_import() (else parentless_not_allowed or PERMISSION_DENIED); - delta sorted/unique/limit/paths; - Put blobs live with matching size (missing_blob, size_mismatch); - REDACTED for quarantined or purged Put ids, while entries carried over unchanged need no check; - delete/put versus T1 (delete_absent, delete_not_file, noop_put, file_directory via apply_delta); - authz.can_write on every delta path (PERMISSION_DENIED); - Meta fields within max_message_bytes (TOO_LARGE). Then apply_delta, the expected_root_tree check (root_tree_mismatch), and Meta with creator = who.id_string() (a client never supplies creator). Persist per 5.3, passing checkpoints create.after_validate/after_trees/after_meta/after_derive/after_commit: 1. store new trees and the meta via ObjectRepo.put_encoded; 2. call the EagerHook with EagerInput; 3. compute generation/skip with compute_commit_graph; 4. build object_paths rows for every Put blob at its path and every new tree at its directory path (path_index); 5. one commit_changeset bundle. Identical requests return the same id and never error, both sequentially and concurrently. create_internal (used by rebase and merge) skips client-facing authz and accepts an existing meta id. bootstrap is idempotent: ensure the root changeset (empty root tree; Meta author 'scion <scion@localhost>', time 0/0, creator = config.bootstrap_principal, message 'root'; change_id = BLAKE3('scion-root-v1')[0..16]) and ref main at version 1 via cas_ref(Absent); an existing ref is fine. Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "19.2: property test, for random trees T1 and random valid deltas: apply_delta's resulting tree, flattened, equals scion-refmodel apply_delta, and file_diff(T1, result) == delta (proptest_cases(2_000); #[ignore] full_ 10^5). Each subcode has a failing test: missing blob, size mismatch, noop put, delete absent, delete of a directory, file/dir without deletes (both directions), each path rule, each limit. Deleting the last file in a directory removes the directory; the root may become empty. File-to-directory and directory-to-file transitions with the required deletes succeed. A merge changeset's delta is validated against the first parent only. Identical CreateChangeset requests, sequential and 50-way concurrent, return one id and leave one row. expected_root_tree mismatch. creator is the server-set principal. 19.3: arm Crash at each create.* checkpoint and at meta.commit_changeset.before; afterwards fsck finds no violation and no visible changeset exists, and a retry returns the same id. 19.13: a Put of a redacted (quarantined) blob fails REDACTED; a changeset not touching a redacted file in the parent tree succeeds. 19.18 unit gate: on shared_subtree_repo at 10^3 and 10^5 (10^7 in #[ignore] full_), a 1-file CreateChangeset reads at most depth+2 trees and writes at most depth+1 trees (Counters deltas), independent of size. bootstrap is idempotent.", "size": "L" }, { "id": "P13", "title": "scion-history: commit-graph queries (IsAncestor, MergeBase, first-parent, log iteration)", "wave": 2, "spec_sections": "9.2 commit_graph, 18.5, 8.4 (stack detection support), 11.8 (~n, ^k), 7.4 (Log without path), 19.6 (commit_graph)", "owns": [ "crates/scion-history/src/graph.rs", "crates/scion-history/tests/graph_*.rs" ], "depends_on": [ "P01", "P03", "P07" ], "deliverables": "Implement the P13 part of PLAN 2.6 in graph.rs, as impl History blocks. Read ChangesetRows (generation, skip, parents decoded from the encoded changeset) through ObjectRepo/MetaStore; this package does not depend on create_changeset. is_ancestor follows 18.5 (generation-pruned DFS; a is an ancestor of itself). is_first_parent_ancestor uses skip pointers in O(log n). merge_bases returns ALL best common ancestors (common ancestors not dominated by another common ancestor), sorted by id, using generation-ordered search. nth_first_parent and nth_parent implement '~n' and '^k'. log_page walks newest-first by generation then id (deterministic), optionally first-parent only, with an opaque page token (an encoded frontier) giving stable paging. Tests build DAGs by inserting ChangesetRows directly into MemMetaStore, using P00's compute_commit_graph, and compare with scion-refmodel MRepo. Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "19.6 commit_graph: generation and skip pointers equal the refmodel for random DAGs (including octopus merges, multiple roots and criss-cross). IsAncestor, is_first_parent_ancestor and MergeBase agree with the refmodel on random queries (10^4 by default; #[ignore] full_ 10^5). A criss-cross gives 2 bases. log_page paging is stable and complete (concatenated pages == the full walk). ~n and ^k edge cases (beyond root gives None).", "size": "M" }, { "id": "P14", "title": "scion-derived framework: registry, engine, lazy iterative derivation, leases, generic query path", "wave": 2, "spec_sections": "9.1, 5.2 (derived, derive_leases), 5.3 (eager rows), 13.5 (purge of content-bearing records), 19.6 (framework items)", "owns": [ "crates/scion-derived/src/{registry.rs,engine.rs,ctx.rs,lease.rs}", "crates/scion-derived/tests/framework_*.rs" ], "depends_on": [ "P01", "P03" ], "deliverables": "Implement the P14 part of PLAN 2.7. Registry: register and look up by name and version; v1_default registers the commit_graph and path_index descriptors (computed by History), history and git_mapping as eager types (constructing scion_derived::history::HistoryType and git_mapping::GitMappingType, whose behaviour is implemented by P21), and blame as lazy (blame::BlameType, P24). latest_version; content_bearing_types. StoreDeriveCtx implements DeriveCtx over ObjectRepo: blob_or_empty returns empty for quarantined or purged blobs; meta returns None when redacted; put_object uses ObjectRepo.put_encoded. DerivedEngine: - impl EagerHook runs every registered eager type and concatenates their rows. required_eager lists eager (name, version) pairs for fsck. - get(ty, version, key) returns the stored value or derives lazily. The lazy algorithm uses an explicit work stack (NEVER recursion): find the nearest keys whose values are stored by walking deps(), then derive forward in dependency order, writing back every intermediate value with put_derived (ON CONFLICT DO NOTHING). - Leases (try_lease/release_lease) only deduplicate concurrent work. A derivation proceeds even if the lease is not acquired, after a short wait-and-recheck. - Version selection: None means the latest registered version; old versions remain readable. - get_stored. purge_content_for deletes records of content-bearing types whose source_object is the id. - Reports derive duration and backlog through the DeriveMetrics hook set via set_metrics (no scion-obs dependency). Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "19.6: two workers deriving the same key concurrently produce byte-identical values and exactly one row (test-only lazy type). A lazy derivation over a chain of 10^5 dependencies (default 10^4; #[ignore] full_ 10^5) completes on a thread with a 256 KiB stack (no recursion growth) and writes back every intermediate value. A version bump re-derives on query while the old version's values stay readable. A test-only type registered at runtime is queryable through DerivedEngine::get (the generic path) with no other change. Lease loss does not affect correctness. purge_content_for deletes only content-bearing records keyed by the id. The EagerHook output contains the rows of all registered test eager types.", "size": "M" }, { "id": "P15", "title": "Gated real backends: scion-store-crdb (CockroachDB) and scion-store-s3 (S3/MinIO)", "wave": 2, "spec_sections": "3.5, 5.1, 5.2, 5.3, 5.4, 16.1 classes 1/2, 19.3 (on real backends)", "owns": [ "crates/scion-store-crdb/src/**", "crates/scion-store-crdb/tests/**", "crates/scion-store-s3/src/**", "crates/scion-store-s3/tests/**" ], "depends_on": [ "P01", "P03" ], "deliverables": "Implement PLAN 2.4 following docs/contracts/storage-semantics.md exactly. CRDB: embedded MIGRATIONS create every SPEC 5.2 table keyed by repo_id first, plus audit_log and the secondary indexes listed in the contract; migrate is idempotent. Each MetaStore method runs in one SERIALIZABLE transaction via deadpool-postgres and tokio-postgres, with a client-side retry loop on SQLSTATE 40001 (bounded, with jitter; when exhausted, return UNAVAILABLE/txn_retry). Inserts use ON CONFLICT DO NOTHING. cas_ref runs UPDATE ... WHERE name=$1 AND version=$2 (or INSERT for Absent and DELETE for delete), inserts the ref_log, inserts landed rows (a unique violation 23505 rolls back and returns LandedConflict with the existing row) and inserts the audit row, all in the same transaction. Consistency::Stale uses AS OF SYSTEM TIME follower_read_timestamp(). changesets_by_prefix uses a range scan on the id column. Connection errors map to UNAVAILABLE/store_unavailable. connect_fresh_for_tests creates a unique database. S3: object_store AmazonS3 with endpoint, path-style and allow_http for MinIO; get on a missing key returns None; copy and delete are idempotent; list uses prefix listing. Keys are prefixed with key_prefix (unique per test run). Every test is gated: when SCION_TEST_CRDB_URL or SCION_TEST_S3_ENDPOINT is unset it prints 'SKIP: ...' and passes. Test names start with external_. Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "When the env vars are set: scion_store::conformance::meta_store_suite against CrdbMetaStore and blob_store_suite against S3BlobStore. Also: migration idempotency; 40001 retry exercised by two conflicting concurrent transactions; Stale reads may miss a just-committed row while Strong reads see it; 1000-way concurrent cas_ref yields exactly one winner. 19.3 on real backends: ObjectRepo over CRDB plus S3 with InstrumentedBlob faults (put fails) leaves no objects row; FailAfter on meta.insert_object leaves a durable row and bytes. When unset, every test prints SKIP and passes.", "size": "L" }, { "id": "P16", "title": "scion-daemon: Unix-socket caching tier, local authorization, redaction tracking, verbatim forwarding", "wave": 2, "spec_sections": "3.3, 6.3, 7.3 (daemon semantics), 7.8, 10.1-10.5, 13.3 (daemon), 16.1 classes 1/6/7, 17 (socket permissions), 18.8 (daemon loop), 19.8, 19.9, 19.11 (daemon), 19.12 (daemon), 19.13 (daemon)", "owns": [ "crates/scion-daemon/src/**", "crates/scion-daemon/tests/**" ], "depends_on": [ "P01", "P04", "P06", "P08", "P10" ],Lines 1–311 of 661. Load more
[ { "id": "P00", "title": "Foundation: workspace, frozen API stubs for all crates, scion-core, scion.v1 proto, contract docs, CI scripts", "wave": 0, "spec_sections": "3.2, 3.4, 4.1-4.3 (type shapes), 4.2 (ids and text form), 5.2 (row types), 6.5 (Limits), 7.1-7.8 (complete scion.v1 proto), 7.2 (error table), 8.3 (line-diff3-v1 contract), 9.2 (commit_graph formula), 18.5, 19.0 (fault registry, checkpoint names), 20.1", "owns": [ "Cargo.toml", "Cargo.lock", "rust-toolchain.toml", "rustfmt.toml", "clippy.toml", ".cargo/config.toml", ".gitignore", "crates/*/Cargo.toml (all 23 crates)", "crates/*/src/lib.rs (all 23 crates)", "crates/scion-proto/build.rs", "proto/scion/v1/scion.proto", "crates/scion-core/**", "crates/scion-history/src/{hooks.rs,config.rs,types.rs,graph_compute.rs}", "crates/scion-derived/src/types.rs", "crates/scion-server/src/{grpc.rs,handlers/mod.rs,authz/mod.rs}", "crates/scion-wc/src/points.rs", "initial stub content of every other module file listed in PACKAGES.json (ownership of each transfers to the listed package)", "scripts/{check.sh,check-layers.sh,layers.toml,test-external.sh}", "docker/compose.test.yml", ".github/workflows/ci.yml", "docs/contracts/{line-diff3-v1.md,storage-semantics.md,amendments.md}", "docs/implementation-defined.md", "docs/testing.md", "testdata/README.md" ], "depends_on": [], "deliverables": "Create the whole Cargo workspace so that it compiles with every public API of docs/PLAN.md section 2 present. Concretely: (1) Root Cargo.toml: resolver = 3, edition 2024, an explicit members list of the 23 crates in PLAN 1.2. It has [workspace.package] and [workspace.lints] (rust unsafe_code = deny; clippy unwrap_used/expect_used/panic = deny, with clippy.toml allow-unwrap-in-tests/allow-expect-in-tests = true). It has [workspace.dependencies] pinning EXACT versions, resolved at scaffold time, of every third-party crate in PLAN 5.3, plus the profiles in PLAN 3.5. Generate and commit Cargo.lock. rust-toolchain.toml pins the installed 1.98 toolchain with rustfmt and clippy. (2) Every crate Cargo.toml declares all internal deps from PLAN 1.2 and all third-party deps it will need (PLAN 5.3 column 'used by'), dev-deps included, so later packages never edit manifests. It also declares features: scion-server/testing, scion-daemon/testing, scion-wc/testing, scion-import/testing, scion-auth/test-hooks. Bins: scion-server, scion-daemon, scion (crate scion-cli), scion-import; each src/main.rs is a stub. (3) Fully implement scion-core exactly as PLAN 2.1, with unit tests: error table, ids and text form, empty_tree_id, Mode, conflict types, Principal, Tier, Clock/ManualClock, Limits defaults, FaultRegistry semantics, points, trace, test_support. (4) Write proto/scion/v1/scion.proto completely per PLAN 2.13 (all six services, every message). crates/scion-proto/build.rs uses tonic-prost-build (protoc is on PATH) and emits a file descriptor set. The v1 module and FILE_DESCRIPTOR_SET are real; the other scion-proto modules are stubs. (5) Every other crate gets lib.rs and the module files named in PACKAGES.json owns lists. Each holds every PLAN section 2 item with rustdoc that cites the SPEC section. Bodies follow PLAN 2.0: todo!(\"<owning package id>\") for behaviour; real assembly constructors and accessors (History::new, ObjectRepo::new, Storage::new, Storage::in_memory, MemMetaStore::new, MemBlobStore::new, DerivedEngine::new, StoreDeriveCtx::new, Registry::empty, Redactor::new, ObjectAuthorizer::new, RedactionSet::new); real trivial impls of NoEager and AllowAll; real compute_commit_graph (9.2: generation = 1 + max parent generation, root 1; skip[k] = first-parent ancestor at distance 2^k); no-op handlers::admin::spawn_background. Handler stubs return Err(Status::unimplemented(\"Pxx\")). scion-server/src/grpc.rs is real: it implements the six generated tonic server traits, and each method delegates in one line to the handlers::<file>::<rpc> function listed in PLAN 2.14. (6) docs/contracts/line-diff3-v1.md (normative, PLAN 6.3): line splitting. Myers forward greedy O(ND) with the exact k-selection rule: move down (insert) when k == -d or (k != d and V[k-1] < V[k+1]), else move right (delete). Snake extension, storage of V per d, backtracking with the same rule, no heuristics, no cost cap. Edit-script normalization (within a changed run, deletions are listed before insertions). Hunk extraction. diff3 region building (transitive closure of b->o and b->t hunks whose base ranges overlap or touch, where an empty insertion range touching or inside another range counts as touching). A one-sided region applies. A two-sided region applies once if both sides produce identical text, otherwise it conflicts. Result assembly. At least 12 hand-worked examples, including tie-sensitive ones (swap AB/BA, repeated lines, empty inputs, unterminated last line, CRLF). (7) docs/contracts/storage-semantics.md: for every MetaStore method, the transaction boundary, reads, writes, uniqueness/conflict behaviour, outcome mapping, the CockroachDB DDL/SQL shape (all SPEC 5.2 tables, plus audit_log and secondary indexes: changeset id prefix, derived source_object, redactions epoch, redactions purge_after) and notes on the Mem implementation. Also ObjectRepo write ordering and the key layout. (8) docs/contracts/amendments.md (protocol plus an empty log). docs/implementation-defined.md: transcribe PLAN section 6 verbatim as the documented implementation-defined behaviours. docs/testing.md: summary of PLAN section 3 with env vars and commands. testdata/README.md. (9) Scripts. scripts/check.sh runs fmt check, build, test, clippy -D warnings and check-layers; --full sets SCION_FULL=1 and runs cargo test --release -- --include-ignored. scripts/check-layers.sh plus scripts/layers.toml implement PLAN 1.3 from cargo metadata, with a --self-test mode proving an injected upward edge is rejected. scripts/test-external.sh. docker/compose.test.yml: cockroachdb single-node insecure on 26257 and MinIO on 9000, with bucket creation. .github/workflows/ci.yml: check.sh on push; nightly --full and external jobs. (10) Before finishing, self-review against SPEC.md and PLAN section 2. cargo doc --workspace --no-deps must be warning-free. Gate G0 (a fresh-agent contract review) follows this package. Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "scion-core unit tests: ErrorCode::ALL maps to the exact SPEC 7.2 gRPC status and retryable flag (19.7). ObjectId text form round-trips for every kind; ids with equal digests and different kinds are unequal and print differently (19.1). parse_text rejects bad input. ChangeId hex round-trips. empty_tree_id equals BLAKE3 of [0x01,0x00,0x00] with kind Tree. Principal parse/display. FaultRegistry: arm with times, hits counted when unarmed, Crash latch until reset, Delay. compute_commit_graph on hand-built rows (root generation 1; skip pointers at distances 1, 2, 4, 8 on a 10-chain; merge generation = 1 + max). Limits::default matches SPEC 6.5. scripts/check-layers.sh passes, and --self-test proves an upward edge is rejected. cargo build, clippy and doc are clean.", "size": "XL" }, { "id": "P01", "title": "scion-object: canonical codec, entities, path/ref/revision grammar, golden vectors", "wave": 1, "spec_sections": "4.1, 4.2 (ids inside objects), 4.3.1-4.3.4, 4.3.6 (ref name grammar), 4.4, 9.2 (HistoryNode encoding, derived kind 0x10), 11.8 (revision grammar parsing), 17 (decoded-form validation)", "owns": [ "crates/scion-object/src/{enc.rs,canonical.rs,extensions.rs,path.rs,tree.rs,meta.rs,changeset.rs,history_node.rs,refname.rs,revision.rs,flat.rs}", "crates/scion-object/tests/**", "testdata/golden/objects/**" ], "depends_on": [ "P00" ], "deliverables": "Implement PLAN 2.2 completely. Encoder/Decoder: uvarint is minimal LEB128 of at most 10 bytes, with overflow rejected; svarint is zigzag; bytes/list lengths may not exceed the remaining input; option tag must be 0 or 1; finish() rejects trailing bytes. All errors are INVALID_ARGUMENT with the scion-core decoding subcodes. Extensions: strictly ascending unique tags. Unknown odd tags are rejected (unknown_critical_extension). Even tags are preserved byte-for-byte; v1 has no known odd tags. Tree: format_version must be 1 (else unknown_format_version). Entries are strictly ascending by bytewise name and unique; names obey 4.4. Modes 1-3 require a Blob id, mode 4 requires a Tree id with size 0. validate_non_root rejects an empty tree (empty_tree). Meta: fields in SPEC order; Timestamp is a pair of svarints. Changeset: parents of kind Changeset, unique; root_tree of kind Tree; meta of kind Meta; change_id is 16 raw bytes; delta strictly ascending by path, op 1 = Put (mode 1..=3, id of kind Blob, size) and 2 = Delete. validate_shape(limits) checks parent count/uniqueness, delta sort/uniqueness, max_delta_entries and path rules (TOO_LARGE for limits, with a Limit payload). HistoryNode: format_version 1, kind u8 (1 File, 2 Dir), content, changeset, parents list, children list<(name, id)> ascending (must be empty for File), extensions. KIND = Derived(0x10). Name/RepoPath: 1-255 bytes, no NUL or '/', not '.' or '..', not '.scion' under ASCII case-insensitive comparison; paths are at most 4096 bytes with no leading/trailing '/' and no empty component; the root is the empty path; ancestors() and is_ancestor_of are component-wise. RefName: ^[a-z0-9][a-z0-9._/-]{0,199}$, with no '..', no '//' and no trailing '/'. parse_revision covers: full id text; 'c1' plus 8..63 hex (IdPrefix); a bare name, treated as a ref name or a bare hex prefix of at least 8 digits, decided later by the server; '<ref>@<u64>'; 'git:<40 hex>'; and chains of '~n' / '^k' suffixes (a bare '~' or '^' means 1; ^0 is invalid). flat::file_diff implements SPEC 4.3.4 FileDiff over FlatTree. Golden vectors: testdata/golden/objects/*.json with at least 14 cases: empty tree; a tree with all four modes; a tree with an even extension; a Meta with and without committer; a Meta with extension tag 2; the root changeset; a changeset with 2 parents and mixed Put/Delete; a changeset with an even extension; HistoryNode File and Dir; a blob. Each file is {name, kind, description, hex, id_text}. A test re-encodes every case from its constructor in Rust and compares bytes and id. A second test asserts a BLAKE3 digest constant over the sorted golden file contents, so any edit to the files fails CI (19.1 freeze). Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "19.1: golden vectors frozen (encode equality plus digest constant). Round trip decode(encode(x)) == x and encode(decode(b)) == b with proptest over random valid Tree/Meta/Changeset/HistoryNode. The default count is proptest_cases(10_000); a #[ignore] full_ test runs 10^6. One rejection test per class: non-minimal varint, 11-byte varint, length overrun, trailing bytes, unsorted entries, duplicate entries, unknown format_version, unknown odd extension tag, invalid mode, mode/kind mismatch. An unknown even tag is preserved byte-for-byte and changes the id. Equal digests with different kinds are never confused (a Tree id is rejected where a Blob id is required). 19.2 path-rule violations at parse level: '.', '..', '/', NUL, '.scion'/'.SCION'/'.Scion', 256-byte name, 4097-byte path, leading/trailing '/'. Ref-name grammar table. Revision grammar parse table (at least 25 cases, valid and invalid). file_diff unit cases (file<->dir transitions produce Delete plus Put).", "size": "M" }, { "id": "P02", "title": "scion-merge: normative Myers diff and line-diff3-v1 engine", "wave": 1, "spec_sections": "8.3 (content merge, binary detection, decision table), 9.2 (blame line mapping), 7.4 (Diff hunks), 11.7 (marker format), 19.4", "owns": [ "crates/scion-merge/src/{split.rs,myers.rs,hunks.rs,merge3.rs,binary.rs,classify.rs,markers.rs,origins.rs,unified.rs}", "crates/scion-merge/tests/**", "testdata/golden/diff/**" ], "depends_on": [ "P00" ], "deliverables": "Implement PLAN 2.5 exactly according to docs/contracts/line-diff3-v1.md. That document is normative; if it is ambiguous, follow its worked examples and file an amendment. Do not use any third-party diff crate in src (similar is allowed only as a dev-dependency to cross-check edit-script minimality). split_lines keeps terminators. myers is the forward greedy O(ND) algorithm with the contract's k-selection rule and backtracking. diff_lines groups non-equal runs into hunks. is_binary: a NUL within the first 8000 bytes. merge3: region building, one-sided apply, identical two-sided apply-once, conflicts reported as ConflictHunk line ranges (base/ours/theirs, u32), and byte-exact output (no CRLF normalization, unterminated final lines preserved). classify implements 8.3 rows 1-6 in order: o==b take t; t==b take o; o==t take o; b absent with o and t present and different gives AddAdd; exactly one side absent and the other differing from b gives ModifyDelete; any side a symlink with sides differing gives Symlink; otherwise NeedContent. merge_mode applies the first three rows to modes. render_markers writes the exact 11.7 marker lines. line_origins maps child lines to parent lines via diff_lines (for blame). unified_hunks produces unified-diff hunks with context. Produce at least 60 golden vectors in testdata/golden/diff/*.json: diff cases {a_hex, b_hex, edits} and merge cases {base_hex, ours_hex, theirs_hex, result_hex | conflicts}. Include all worked examples from the contract doc and tie-break-sensitive inputs. Freeze them with a digest-constant test. Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "19.4 content level: o==b => t; t==b => o; o==t => o; non-overlapping, non-adjacent hunks from both sides both apply; overlapping hunks conflict; adjacent hunks (zero unchanged lines between) conflict; identical hunks apply once. Each property runs with proptest_cases(10_000), with #[ignore] full_ variants at 10^6. Binary boundary: NUL at byte 7999 is binary, at byte 8000 is text. CRLF and unterminated-final-line merges are byte-exact. classify has one test per 8.3 table row. Myers optimality: edit length equals an O(N*M) LCS-based minimum on small random inputs. Determinism: the same input always gives the same output. Golden vectors are frozen. RECOMMENDED: a differential against `git merge-file -p` on clean text merges (skipped when git is absent), with disagreements recorded in crates/scion-merge/tests/GIT_MERGE_FILE_DIFFERENCES.md.", "size": "M" }, { "id": "P03", "title": "scion-store: in-memory MetaStore/BlobStore, ObjectRepo, counters and fault decorators, fsck, backend conformance suites", "wave": 1, "spec_sections": "5.1, 5.2, 5.3 step 1, 5.4, 4.5 (fsck), 9.1 (leases), 13.2 (redaction transactions and byte moves), 16.1 classes 2/3/7, 17 (audit log, decoded-form validation), 19.0 (counters, fault hooks), 19.3", "owns": [ "crates/scion-store/src/{types.rs,meta.rs,blob.rs,storage.rs,keys.rs,mem_meta.rs,mem_blob.rs,repo.rs,instrument.rs,fsck.rs,conformance.rs}", "crates/scion-store/tests/**" ], "depends_on": [ "P00" ], "deliverables": "Implement PLAN 2.3 and docs/contracts/storage-semantics.md. MemMetaStore: all state is im persistent maps behind one parking_lot::Mutex, and each trait method is one atomic step (strictly serializable). Required behaviour: - ON CONFLICT DO NOTHING for objects, changesets, children, object_paths and derived; commit_changeset inserts every row independently, so a re-commit fills in missing rows. - cas_ref: RefExpect::Absent or Version(v); on success version becomes v+1 (1 on create); ref_log row, landed rows and audit row in the same step; a landed (ref, change_id) unique violation returns LandedConflict and applies nothing; a version mismatch returns the current row. - redact: increments counters.redaction_epoch, inserts rows stamped with the new epoch, sets objects.state = quarantined. unredact: Quarantined only (else INVALID_ARGUMENT/not_quarantined), epoch+1, state live. mark_purged: clears inline_data, state purged. - Leases are TTL-based against the provided now. - A ring of the last K committed snapshots serves Consistency::Stale (stale_lag_commits, set_stale_lag, advance_stale_view). - fail_txn_every: every Nth transaction fails with UNAVAILABLE/txn_retry and no effect. MemBlobStore supports corrupt(), set_down() and keys(). keys.rs implements the SPEC 5.1 layout. InstrumentedMeta/InstrumentedBlob count meta_txns, meta_rows_read, object_paths_rows_read, lease_calls and blob ops/bytes, and check fault points meta.<method>.before / meta.<method>.after and blob.<method>.before / blob.<method>.after (FailAfter means the effect is applied and the call then fails). Leases emit a tracing span with target scion::lock. ObjectRepo: - put_blob/put_encoded/put_verified validate size (TOO_LARGE) and the decoded form of known kinds (Tree/Meta/Changeset/HistoryNode), and reject redacted ids (REDACTED). - Blobs always go to object storage. Tree/Meta/Derived objects at or below inline_threshold_bytes are stored inline, others as zstd frames at zstd_level. - Write order: bytes first (checkpoint upload.after_bytes), then the objects row (checkpoint upload.after_row); a live row therefore always implies durable bytes. - get/get_frame verify BLAKE3 of the decompressed content; on mismatch they record an integrity failure (Counters plus an ERROR log event 'integrity_alert') and return INTEGRITY. - Every read records object reads by kind and bytes. - stat/get use a stale read first and retry a miss with a strong read before NOT_FOUND. - LRU caches of decoded trees, metas and changesets serve immutable reads even while the metadata store is down. - quarantine_bytes copies to the quarantine key and then deletes the live key (checkpoint quarantine.after_copy). restore_bytes reverses it. purge_bytes deletes both keys. fsck walks every changeset and checks SPEC 4.5: digest of every live object, presence of root tree, meta, parents and delta blobs, delta == FileDiff(parent tree, root tree), generation and skip consistency, required eager rows, and ref targets exist. Quarantined or purged bytes are not violations. conformance::meta_store_suite and blob_store_suite are public async fns (always compiled, a test-support API). They cover every method contract, including the concurrency cases below, and run against MemMetaStore/MemBlobStore in this package's tests. P15 reuses them for CRDB/S3. Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "Tests in crates/scion-store/tests: - meta_store_suite and blob_store_suite on Mem. - CAS: 1000 concurrent cas_ref calls at the same expected version give exactly one Committed; versions advance by exactly 1; ref_log has no gaps (19.5, store level). ABA: A->B->A, then a CAS expecting the first version fails. - LandedConflict rolls back the whole cas_ref. - 19.3: a blob-store fault before put leaves no objects row; FailAfter on put still leaves no row unless the row insert ran; a row never exists without bytes. Inline threshold at threshold-1, threshold and threshold+1 stores in the expected place and reads back identically. A stale-read miss is retried strongly (stale_lag) before NOT_FOUND. Orphan bytes (a put with no row) are invisible to stat/get. - 19.1: a blob id is independent of zstd level (1, 3, 19) and framing (a multi-frame encoding of the same content). - Corrupt stored bytes give INTEGRITY and integrity_failures +1 (16.1 class 7). - fail_txn_every leaves no partial state. - Redaction epoch increments atomically with the state changes. - Counters count exactly (tree reads include inline trees). - fsck detects each Violation kind on hand-corrupted state and passes on valid state.", "size": "L" }, { "id": "P04", "title": "scion-config and scion-obs: TOML configs, structured logging, metrics, health", "wave": 1, "spec_sections": "6.1, 6.3, 6.4, 6.5, 15.1, 15.2, 15.3, 16.1 class 7 (integrity alert metric), 19.15, 19.17", "owns": [ "crates/scion-config/src/**", "crates/scion-config/tests/**", "crates/scion-obs/src/**", "crates/scion-obs/tests/**" ], "depends_on": [ "P00" ], "deliverables": "Implement PLAN 2.11 and 2.12; lib.rs and Cargo.toml stay P00's. Config: - serde with #[serde(deny_unknown_fields)] everywhere, wrapped in serde_path_to_error so ConfigError::UnknownKey names the full dotted key (for example 'limits.max_blobs'). - '$NAME' string values are resolved from Env in a pre-pass over toml::Value, keeping the key path. An unresolved variable is ConfigError::UnresolvedVar naming the variable and the key. - Every default from SPEC 6.5 and PLAN 2.11 (including the implementation-defined (+) keys) applies when a key is omitted. - Durations use humantime ('12h', '7d'); millisecond keys are integers. - resolve_endpoint precedence: flag > SCION_ENDPOINT > working-copy config > user config. - for_tests constructors. Obs: - init_logging emits JSON (one event per line) or text. - RpcLogEvent::emit logs the REQUIRED 15.1 fields: request_id, rpc, tier, principal, repo, status, duration_ms, and when applicable changeset_id, ref, ref_version, land_attempt. - path_hash returns 16 hex characters of BLAKE3. Raw repository paths are never logged (PLAN 6.9 log access policy). - Metrics registers every SPEC 15.2 metric with the exact names and labels (scion_rpc_duration_seconds{rpc,tier,code}, scion_cache_requests_total{tier,kind,result}, scion_cache_bytes{tier}, scion_land_attempts{result}, scion_ref_cas_failures_total{ref}, scion_land_duration_seconds, scion_derive_duration_seconds{type,mode}, scion_derive_backlog{type}, scion_redaction_epoch{tier}, scion_redaction_lag_seconds{tier}, scion_policy_epoch_info{epoch}, scion_object_bytes_served_total{tier,encoding}, scion_upstream_requests_total{tier}), plus scion_integrity_failures_total{tier}. NAMES lists them all. - health_router serves /metrics (text exposition), /healthz (200 while alive) and /readyz (200 or 503 from Readiness::check). serve_admin binds it. - capture::LogCapture installs a process-global in-memory JSON layer for tests, with find_needles for log scans. Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "19.15: an unknown key is rejected with the key name, at top level and in a nested section, for server, daemon and client configs. $VAR resolves. An unresolved $VAR errors, naming the variable and key. Every SPEC 6.5 default is applied when its key is omitted (one assertion per row of the 6.5 table, plus the (+) keys). The SPEC 6.1 example file parses with env vars supplied. 19.17 unit level: every 15.2 metric name appears in render() after one observation; /healthz returns 200; /readyz returns 503 then 200 as a toggled Readiness changes; RpcLogEvent produces all required fields as JSON keys; path_hash is stable and never equals the input; LogCapture.find_needles finds a planted secret.", "size": "M" }, { "id": "P05", "title": "scion-policy: acl.toml parsing, evaluation, epoch, hot reload", "wave": 1, "spec_sections": "6.2, 12.4, 4.3.8, 16.1 class 9, 19.12 (policy semantics and reload)", "owns": [ "crates/scion-policy/src/**", "crates/scion-policy/tests/**" ], "depends_on": [ "P00" ], "deliverables": "Implement PLAN 2.9. Parse with toml and serde (deny_unknown_fields; UnknownKey names the key). version must be 1. A '/' rule is REQUIRED (MissingRootRule). Groups: nested groups are resolved; cycles are a GroupCycle error listing the cycle; unknown groups are rejected. Principal forms: 'github:<login>' (compared case-insensitively), 'github-id:<n>' (matches Principal::GitHub{id: Some(n)}), 'service:<name>', 'group:<name>'. Prefixes are written with a leading '/' and normalized (no trailing '/'). Matching is component-wise, so '/secrets' matches secrets and secrets/x but not secrets2. For each path, the longest matching rule governs, independently for read and for write; rules never merge. read_governed_by_root reports whether the governing read rule is '/'. Refs: an exact name beats any pattern; among patterns (a single trailing '*' segment) the longest wins. Admin permissions: redact, purge, refs. Default deny. epoch = first 8 bytes, big-endian, of BLAKE3 over a canonical normalized serialization (sorted groups, rules and principals), so that equivalent files agree. PolicyHandle: arc-swap of Arc<Policy>; load() fails on an invalid or missing file. reload_now swaps atomically only when the new policy is valid. An invalid file keeps the last good policy, records last_error and logs an operator-visible ERROR. A watch channel publishes epoch changes. spawn_watcher uses notify plus a 1 s polling fallback. Tests use reload_now for determinism. allow_all_for_tests() grants everything to everyone. impl From<PolicyError> for ScionError (INVALID_ARGUMENT). Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "19.12 policy semantics: longest prefix wins; component-wise matching ('/secrets' vs '/secrets2'); independent read and write resolution; nested groups; cycle detection; the github-id: form; case-insensitive github logins; ref pattern precedence (exact > longest pattern) and default deny; admin permissions. Reload: a valid change is applied atomically and the epoch changes; an equivalent reformatted file gives the same epoch; an invalid change keeps the last good policy, surfaces last_error and leaves the epoch unchanged; an invalid or missing file at startup fails. A concurrency test: readers never observe a torn policy during repeated reloads.", "size": "M" }, { "id": "P06", "title": "scion-auth: Ed25519 PASETO tokens, keyring rotation, constant-time secrets, GitHub client, device flow, FakeGitHub", "wave": 1, "spec_sections": "12.1, 12.2, 12.3, 17 (secrets never logged), 16.1 class 10, 19.11", "owns": [ "crates/scion-auth/src/**", "crates/scion-auth/tests/**", "crates/scion-testkit/src/fake_github.rs", "crates/scion-testkit/tests/fake_github_*.rs" ], "depends_on": [ "P00" ], "deliverables": "Implement PLAN 2.10 and the FakeGitHub part of PLAN 2.21. Tokens: PASETO v4.public via pasetors (Ed25519). Claims sub, gid, iat, exp. kid goes in the footer as JSON {\"kid\":...}; kid = first 16 hex characters of BLAKE3(public key). Keyring::from_files reads PKCS#8 PEM Ed25519 keys. The first key signs and all keys verify. generate_for_tests and write_pem_files support harnesses. Verifier::verify returns UNAUTHENTICATED for malformed, unknown-kid, bad-signature, tampered and expired tokens (exp <= now). ct_eq wraps subtle::ConstantTimeEq and is the ONLY comparison primitive for secrets; under feature test-hooks it counts calls. ServicePrincipals::verify always computes SHA-256(secret) and calls ct_eq, comparing against a dummy digest for unknown names. Secret has redacting Debug/Display. GitHubClient::user calls GET {api_url}/user with 'Authorization: Bearer'. 401/403 maps to UNAUTHENTICATED; transport errors and 5xx map to UNAVAILABLE. device_flow::start/poll implement the GitHub device flow (POST {oauth_base}/login/device/code, POST {oauth_base}/login/oauth/access_token), handling authorization_pending, slow_down and expired_token. FakeGitHub (axum on 127.0.0.1:0) serves GET /user (tokens registered with add_user), the device-code endpoints (approve_device_code completes the flow) and a set_down switch returning 503. Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "19.11 token items: a valid token verifies with the right sub and gid; expired, malformed, wrong-key, tampered-payload and tampered-footer tokens are rejected. Rotation: a token signed by key 2 verifies while key 2 is listed and fails once it is removed; new tokens are signed by the first key. Service secret: right succeeds, wrong fails, unknown name fails; ct_eq_calls increases on BOTH the right and the wrong path (the constant-time primitive is used on both). GitHubClient against FakeGitHub: valid token maps login and id; invalid token gives UNAUTHENTICATED; FakeGitHub down gives UNAVAILABLE. A device-flow round trip against FakeGitHub, including authorization_pending. Secret Debug output never contains the secret.", "size": "M" }, { "id": "P07", "title": "scion-refmodel: independent reference model (19.0 oracle)", "wave": 1, "spec_sections": "4.1-4.4, 4.3.4 validity, 8.1 (validation outcomes), 8.2, 8.3, 8.4 (sequential land semantics), 9.2 (commit_graph, history TREESAME, blame), 11.2-11.3 (materialization, sparse scope), 13.2 (--all-versions), 13.5 (redacted content as empty)", "owns": [ "crates/scion-refmodel/src/**", "crates/scion-refmodel/tests/**" ], "depends_on": [ "P00" ], "deliverables": "Implement PLAN 2.20 as a deliberately naive, single-threaded oracle. INDEPENDENCE RULES: the only third-party dependency is blake3 (and std). Do NOT read or copy any production crate under crates/ other than this one; work only from SPEC.md and docs/contracts/line-diff3-v1.md. check-layers.sh verifies that there are no scion dependencies. Contents: - Its own canonical encoder (enc) for trees (built recursively from the flat MTree), Meta and Changeset, plus id text. - apply_delta, which flattens, applies and rebuilds, returning the 8.1/18.2 subcode strings on invalid deltas (delete_absent, delete_not_file, noop_put, file_directory). - file_diff. - Its own Myers and diff3 written from the contract doc, using the simplest correct code, with no optimizations. - merge_trees implementing the full 8.3 table, both structural file_directory rules, the binary and merge_max_text_bytes rules, and redacted conflicts. - MRepo: commit graph (generation, skip pointers by naive walk), is_ancestor (full BFS), merge_bases (full ancestor-set intersection minus dominated), rebase per 8.2 (fast path, EMPTY), log_path with TREESAME semantics equivalent to the 9.2 history node-reuse rule (follow parents[0] when the content equals parents[0]'s, otherwise all parents where the path exists), blame per 9.2 (first parent, then subsequent parents, remaining lines to the commit; redacted content empty), all_versions (distinct contents ever held). - land_sequential. - materialize(sparse) per 11.2. O(n^2) algorithms are acceptable. Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "Self-tests on hand examples taken from the SPEC tables: every 8.3 row and both structural rules; the 18.2 apply_delta subcodes; root changeset rules; generation and skip on a hand DAG including an octopus merge; a criss-cross DAG giving two merge bases; the TREESAME log on a merge that keeps the first parent's version; blame on a 3-commit example; all worked examples of docs/contracts/line-diff3-v1.md reproduced exactly; materialize with nested sparse prefixes. Property: file_diff(t, apply_delta(t, d)) == d for random t and random valid d (proptest_cases(5_000)). Cross-checks against production code are done later by P17 and other packages.", "size": "L" }, { "id": "P08", "title": "scion-proto: conversions, error mapping, epoch trailer layer, ScionClient, RawCodec; testkit FakeUpstream and TestTls", "wave": 1, "spec_sections": "7.1, 7.2, 7.3-7.7 (message mapping), 7.8 (verbatim forwarding codec), 3.3, 17 (TLS), 19.7 (error mapping, forward compatibility)", "owns": [ "crates/scion-proto/src/{convert.rs,status.rs,trailers.rs,client.rs,endpoint.rs,raw.rs}", "crates/scion-proto/tests/**", "crates/scion-testkit/src/{fake_upstream.rs,tls.rs}", "crates/scion-testkit/tests/upstream_*.rs" ], "depends_on": [ "P00" ], "deliverables": "Implement the Rust surface of PLAN 2.13 (the proto file and codegen are P00's and frozen), plus FakeUpstream and TestTls from PLAN 2.21. convert: From/TryFrom between v1 messages and scion-core/scion-object/scion-store types (ObjectId bytes, EntryInfo, DeltaEntry, MetaFields, ConflictReport, LandedEntry, RefRow/RefLogRow, Limits, ServerInfo pieces). A missing required field or a malformed id gives INVALID_ARGUMENT/missing_field or bad_id_text. Unknown fields are ignored (prost default). status: status_from_error sets the tonic code from ErrorCode::grpc_code and the message, and puts the ErrorDetail encoding (code string, retryable, message, payload oneof) into Status details. error_from_status inverts it; a Status without details maps from its gRPC code, defaulting to UNAVAILABLE. trailers: TRAILER_* constants, read_epochs/write_epochs. EpochTrailerLayer is a tower Layer over http Request/Response bodies. It appends scion-policy-epoch, scion-redaction-epoch and scion-tier to the HTTP/2 trailers frame of EVERY response: unary, server-streaming, client-streaming, error, and trailers-only responses (where tonic puts grpc-status in the headers, add the three keys to those headers). Epoch values are decimal u64. endpoint: Endpoint::parse accepts 'unix:///abs/path' and 'https://host:port'. client: ScionClient::connect uses tonic Channel over a UnixStream connector (hyper-util) or https with rustls (CA from TlsOptions, otherwise native roots). It installs a bearer-token interceptor and a response layer capturing the last epochs from trailers (and trailers-only headers), and keeps an atomic call counter (calls/reset_calls). raw: RawCodec is a tonic Codec whose Encode and Decode are both bytes::Bytes, for byte-level forwarding by the daemon. TestTls::generate creates an rcgen CA plus a server certificate for localhost and 127.0.0.1. FakeUpstream: an in-memory tonic server on 127.0.0.1:0 (TLS optional, via TestTls) implementing: ObjectService (GetServerInfo with configured public keys; GetObjects with per-item ok/not_found/denied/redacted, zstd or raw frames, offset/length ranges, access_class, kill_stream_after_bytes; HasObjects; ResolveRef); QueryService GetChangesets and Resolve; AuthService CheckAccess and WhoAmI; AdminService ListRedactions. Every other method returns UNIMPLEMENTED. The principal comes from the bearer token via the configured principal_of closure. The server is wrapped in EpochTrailerLayer using set_epochs values (tier server). It has set_down, corrupt, redact and deny knobs, and records every request (method, ids, principal). scion-proto tests must not dev-depend on scion-testkit (dependency rule PLAN 1.2); proto-level transport tests use rcgen directly. Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "19.7: every ErrorCode round-trips through status_from_error/error_from_status with the correct tonic code and retryable flag, and every payload variant round-trips. A request with unknown fields (hand-encoded extra tag) decodes. Missing required fields in CreateChangesetRequest/MergeRequest give INVALID_ARGUMENT/missing_field. Property round trip for conversions. Transport tests (crates/scion-proto/tests, rcgen certificates, a minimal ObjectService implementation defined in the test): over TLS and over a Unix socket, ScionClient sees all three trailers on a unary success, a unary error, a server-streaming response, a client-streaming response and a trailers-only error, and last_epochs is updated; the call counter counts. RawCodec forwards a unary and a streaming call byte-identically through a test proxy. FakeUpstream tests (crates/scion-testkit/tests/upstream_*.rs): batch mixing ok/not_found/denied/redacted; range read; stream kill and resume by offset; requests are recorded.", "size": "L" }, { "id": "P09", "title": "scion-testkit corpus: generators, shared-subtree repos, edge cases, vendored git bundles, linearizability checker", "wave": 1, "spec_sections": "19.0 (Corpus, History Recorder), 19.5 (linearizability checker), 19.14 (real git histories), 19.18 (equal-depth generated repos)", "owns": [ "crates/scion-testkit/src/{corpus.rs,shared.rs,edge.rs,encode.rs,git.rs,lincheck.rs}", "crates/scion-testkit/tests/corpus_*.rs", "crates/scion-testkit/tests/lincheck_*.rs", "corpus/git/**" ], "depends_on": [ "P00" ], "deliverables": "Implement PLAN 2.21 except fake_upstream.rs and tls.rs (P08) and fake_github.rs (P06). corpus::generate(CorpusSpec): a seeded (rand_chacha) generator of GenRepo histories of structural MTree snapshots, with configurable file count, depth, fanout, history length, merge ratio, octopus ratio, binary ratio and symlink ratio, plus criss-cross shapes. Text edits are line-level, so merges are meaningful; output is deterministic per seed. shared::shared_subtree_repo(files, depth): a balanced tree of identical subtrees reaching 10^3, 10^5 or 10^7 logical files at the SAME depth (fanout chosen per size, depth fixed), perturbable along one path. encoded_objects() returns only the O(depth x fanout) distinct objects. edge::edge_cases(): empty root, deep path at the 4096-byte limit, 255-byte names, case-only-differing names, file<->dir flips, CRLF, NUL content (binary), executable flips, symlink retargets, delete-and-readd, unterminated last lines. encode::encode_tree uses scion-object to produce production-encoded trees and blobs from an MTree. It is exercised first by P17; this package must not run scion-object code in its own tests, because P01 is in the same wave. git: vendor at least two REAL open-source git histories as bundles under corpus/git/ (candidates: rust-lang/cfg-if and dtolnay/itoa, both MIT OR Apache-2.0 and small with merge commits). Verify license and size (each at most 5 MB, full history via `git bundle create <f> --all`), and add corpus/git/README.md plus LICENSE-<repo> files recording source URL, commit and license. Also build and commit synthetic edge bundles with the git CLI: an octopus merge, multiple roots, empty commits, mode 100664, symlinks, a gitlink, a merge keeping the first parent's version, and rename-free line edits for blame. Tests must then not require the git CLI; bundles(), unpack() (via gix) and GitFixture (which needs git and skips when it is absent) are provided. If network access is impossible, stop and report instead of substituting synthetic history for the real-repository requirement. lincheck: HistoryRecorder (thread-safe, monotonic timestamps) and check_ref_registers, a WGL linearizability search with memoization over a versioned-register model per ref. Read returns (version, target); Cas succeeds iff its expect equals the current version; Land succeeds with version+1. An Unknown completion may or may not have taken effect (crash or timeout). Histories are partitioned per ref; document the cap of about 200 operations per history. Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "Generators are deterministic per seed and realize the requested shapes (octopus, multiple roots, criss-cross detected structurally). The shared-subtree repo has the same depth at 10^3, 10^5 and 10^7 files, and its distinct structural subtree count is O(depth x fanout) (computed structurally). Edge-case catalog sanity. lincheck accepts known-linearizable histories (sequential; concurrent with overlap; with Unknown completions) and rejects known violations (stale read after a completed CAS, two successful CAS at the same version, a version gap) on hand-built cases. Bundles exist, their README and licenses are present, and the size limits hold (no git CLI required).", "size": "L" }, { "id": "P10", "title": "scion-cache: verified disk cache, coalescer, upstream limiter, ref/redaction/access caches", "wave": 1, "spec_sections": "10.2, 10.3, 10.4 (decision caches), 13.3 (daemon redaction set and staleness), 16.1 classes 6/7, 19.9", "owns": [ "crates/scion-cache/src/**", "crates/scion-cache/tests/**" ], "depends_on": [ "P00" ], "deliverables": "Implement PLAN 2.15. DiskCache layout: <dir>/objects/<kind_letter>/<hex[0:2]>/<hex>, holding the payload as received (a zstd frame or raw bytes, with a 1-byte header distinguishing them). Insert writes a temp file in the same directory, verifies BLAKE3 of the decompressed content against the id (INTEGRITY, nothing cached, on mismatch), fsyncs and renames. open() deletes leftover temp files and rebuilds the size and access index by scanning. get verifies on read when verify_on_read is set; a mismatch deletes the entry and returns Ok(None) so the caller refetches. Approximate LRU by last access time: after an insert that pushes usage over max_bytes, evict down to low_watermark x max_bytes. usage() is exact. Coalescer is single-flight per key: concurrent callers share one future, errors are shared, and the entry is removed after completion. UpstreamLimiter: semaphore(max_concurrency) plus a bounded waiter count; overflow gives RESOURCE_EXHAUSTED immediately. RefCache: TTL-bound, and never returns a lower version than previously returned for a ref, even after TTL expiry or out-of-order upstream answers. LocalRedactions: id set, epoch, last refresh and is_stale. AccessCache: entries keyed by (principal, object) with TTL, plus a per-principal root-reader flag with TTL. Decisions never leak across principals, and on_policy_epoch(epoch) clears everything when the epoch differs. Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "19.9 unit level: a corrupt insert is rejected and not cached; a bit flip in a cached file is detected on read (verify_on_read), deleted and get returns None; temp files are removed on open; a simulated crash after the temp write and before rename leaves no visible entry; 10000 concurrent get_or_fetch calls for one key run the fetch exactly once; a randomized insert/get workload keeps usage <= max_bytes at all times and evicted ids read back as None; ref TTL is honored and versions stay monotonic across TTL expiry and reordered observations; queue overflow gives RESOURCE_EXHAUSTED. AccessCache isolates principals and clears on an epoch change.", "size": "M" }, { "id": "P11", "title": "scion-wc local layer: safe filesystem, stat-cache index, status engine, ignore, state, lock, credentials", "wave": 1, "spec_sections": "4.4 (path rules, case collisions), 6.4 (credentials), 11.1, 11.4, 11.5, 17 (checkout path safety, credentials 0600), 18.7, 19.10 (status, ignore, path safety), 19.11 (credentials)", "owns": [ "crates/scion-wc/src/{fs_safe.rs,index.rs,status.rs,ignore.rs,state.rs,lock.rs,credentials.rs}", "crates/scion-wc/tests/local_*.rs" ], "depends_on": [ "P00" ], "deliverables": "Implement the corresponding modules of PLAN 2.17. SafeRoot uses rustix openat with O_NOFOLLOW for every path component (a pre-existing symlinked parent is refused, never followed). Writes go to a temp file plus renameat in the target directory, with the exec bit set for Mode::Executable; symlink() creates symlinks; nothing is ever written outside the root. is_case_insensitive probes the filesystem once. validate_entry_path enforces 4.4 and rejects '.scion' in any case. case_collisions uses ASCII case folding, so it is deterministic on any filesystem; checkout uses it whenever the filesystem is case-insensitive or WcOptions.force_case_insensitive is set. StatCache: (size, mtime_ns, ctime_ns, inode, mode) tuples plus written_at_ns, stored as a binary file .scion/index (versioned header) and saved atomically. is_racy means mtime >= written_at. status follows 18.7 over the TreeLookup trait: walk in-scope files, skip .scion and ignored files, skip inaccessible prefixes; hash files (symlinks hash their target) only when the stat tuple changed or the entry is racy; report modified/added/deleted/mode_changed; refresh clean entries; an unmodified redacted placeholder (index redacted flag plus placeholder bytes) is clean, and an edited placeholder is reported in redacted_dirty and modified; count hashed files. IgnoreMatcher uses ignore::gitignore (nested .gitignore files, negation, directory patterns, anchoring) with git semantics. state: WcState and PendingOperation as canonical TOML (sorted keys, deterministic), saved atomically under .scion/state. lock: fs4 advisory non-blocking lock on .scion/lock. credentials: write creates the file with mode 0600 (and its parent with 0700); read refuses a group- or world-readable file with PERMISSION_DENIED and a clear message. Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "19.10 status: randomized edit sequences (create, modify, delete, chmod +x/-x, symlink swap, rename, truncate-and-write, atomic-rename save, edits within the same second as the index write, i.e. racy-clean) match a full-rehash oracle after every step (proptest_cases(500); #[ignore] full_ at 10^4). Clean status hashes only racy or stat-changed files. .gitignore semantics match `git check-ignore` on a corpus of at least 40 patterns (skipped when git is absent), and a pure table of expected results runs always. Path safety: '..', '.scion', '.SCION' and NUL paths are refused; writes through a pre-existing symlinked parent directory are refused and nothing is created outside the root. case_collisions reports collisions (deterministic on any fs; plus a real collision test when the tmp fs is case-insensitive, as on macOS APFS). State and index round-trip and are atomic (a crash simulated mid-save leaves the old file). The lock excludes a second holder. 19.11: the credentials file is created 0600; read refuses 0640 and 0604.", "size": "L" }, { "id": "P12", "title": "scion-history: tree reading, apply_delta, FileDiff, CreateChangeset, bootstrap", "wave": 2, "spec_sections": "4.3.2, 4.3.4, 4.5, 5.3 step 2, 8.1, 9.2 (commit_graph columns, path_index rows), 13.6, 14 (parentless import_mode), 18.1, 18.2, 16.1 class 4, 19.2, 19.3, 19.18 (CreateChangeset gate)", "owns": [ "crates/scion-history/src/{history.rs,tree.rs,apply.rs,diff.rs,create.rs,bootstrap.rs}", "crates/scion-history/tests/create_*.rs" ], "depends_on": [ "P01", "P03", "P07" ], "deliverables": "Implement the P12 part of PLAN 2.6 (History::new and accessors already exist). tree/lookup/changeset go through ObjectRepo caches. apply_delta follows 18.2 exactly: walk only the ancestors of delta paths, deepest first, applying deletes, then rebuilt children, then puts. Subcodes: delete_absent, delete_not_file, file_directory (ancestor file not deleted, or a directory not fully deleted), noop_put. An empty non-root directory is pruned; the root may become empty. Return the new root plus the new non-empty trees with their canonical bytes. file_diff prunes identical subtree ids and supports a scope filter. create_changeset validates in the exact 8.1 order with the first failure returned: - parents exist, are unique and within max_parents (TOO_LARGE); empty parents only when has_root_changeset() is false, or when import_mode is set and authz.can_import() (else parentless_not_allowed or PERMISSION_DENIED); - delta sorted/unique/limit/paths; - Put blobs live with matching size (missing_blob, size_mismatch); - REDACTED for quarantined or purged Put ids, while entries carried over unchanged need no check; - delete/put versus T1 (delete_absent, delete_not_file, noop_put, file_directory via apply_delta); - authz.can_write on every delta path (PERMISSION_DENIED); - Meta fields within max_message_bytes (TOO_LARGE). Then apply_delta, the expected_root_tree check (root_tree_mismatch), and Meta with creator = who.id_string() (a client never supplies creator). Persist per 5.3, passing checkpoints create.after_validate/after_trees/after_meta/after_derive/after_commit: 1. store new trees and the meta via ObjectRepo.put_encoded; 2. call the EagerHook with EagerInput; 3. compute generation/skip with compute_commit_graph; 4. build object_paths rows for every Put blob at its path and every new tree at its directory path (path_index); 5. one commit_changeset bundle. Identical requests return the same id and never error, both sequentially and concurrently. create_internal (used by rebase and merge) skips client-facing authz and accepts an existing meta id. bootstrap is idempotent: ensure the root changeset (empty root tree; Meta author 'scion <scion@localhost>', time 0/0, creator = config.bootstrap_principal, message 'root'; change_id = BLAKE3('scion-root-v1')[0..16]) and ref main at version 1 via cas_ref(Absent); an existing ref is fine. Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "19.2: property test, for random trees T1 and random valid deltas: apply_delta's resulting tree, flattened, equals scion-refmodel apply_delta, and file_diff(T1, result) == delta (proptest_cases(2_000); #[ignore] full_ 10^5). Each subcode has a failing test: missing blob, size mismatch, noop put, delete absent, delete of a directory, file/dir without deletes (both directions), each path rule, each limit. Deleting the last file in a directory removes the directory; the root may become empty. File-to-directory and directory-to-file transitions with the required deletes succeed. A merge changeset's delta is validated against the first parent only. Identical CreateChangeset requests, sequential and 50-way concurrent, return one id and leave one row. expected_root_tree mismatch. creator is the server-set principal. 19.3: arm Crash at each create.* checkpoint and at meta.commit_changeset.before; afterwards fsck finds no violation and no visible changeset exists, and a retry returns the same id. 19.13: a Put of a redacted (quarantined) blob fails REDACTED; a changeset not touching a redacted file in the parent tree succeeds. 19.18 unit gate: on shared_subtree_repo at 10^3 and 10^5 (10^7 in #[ignore] full_), a 1-file CreateChangeset reads at most depth+2 trees and writes at most depth+1 trees (Counters deltas), independent of size. bootstrap is idempotent.", "size": "L" }, { "id": "P13", "title": "scion-history: commit-graph queries (IsAncestor, MergeBase, first-parent, log iteration)", "wave": 2, "spec_sections": "9.2 commit_graph, 18.5, 8.4 (stack detection support), 11.8 (~n, ^k), 7.4 (Log without path), 19.6 (commit_graph)", "owns": [ "crates/scion-history/src/graph.rs", "crates/scion-history/tests/graph_*.rs" ], "depends_on": [ "P01", "P03", "P07" ], "deliverables": "Implement the P13 part of PLAN 2.6 in graph.rs, as impl History blocks. Read ChangesetRows (generation, skip, parents decoded from the encoded changeset) through ObjectRepo/MetaStore; this package does not depend on create_changeset. is_ancestor follows 18.5 (generation-pruned DFS; a is an ancestor of itself). is_first_parent_ancestor uses skip pointers in O(log n). merge_bases returns ALL best common ancestors (common ancestors not dominated by another common ancestor), sorted by id, using generation-ordered search. nth_first_parent and nth_parent implement '~n' and '^k'. log_page walks newest-first by generation then id (deterministic), optionally first-parent only, with an opaque page token (an encoded frontier) giving stable paging. Tests build DAGs by inserting ChangesetRows directly into MemMetaStore, using P00's compute_commit_graph, and compare with scion-refmodel MRepo. Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "19.6 commit_graph: generation and skip pointers equal the refmodel for random DAGs (including octopus merges, multiple roots and criss-cross). IsAncestor, is_first_parent_ancestor and MergeBase agree with the refmodel on random queries (10^4 by default; #[ignore] full_ 10^5). A criss-cross gives 2 bases. log_page paging is stable and complete (concatenated pages == the full walk). ~n and ^k edge cases (beyond root gives None).", "size": "M" }, { "id": "P14", "title": "scion-derived framework: registry, engine, lazy iterative derivation, leases, generic query path", "wave": 2, "spec_sections": "9.1, 5.2 (derived, derive_leases), 5.3 (eager rows), 13.5 (purge of content-bearing records), 19.6 (framework items)", "owns": [ "crates/scion-derived/src/{registry.rs,engine.rs,ctx.rs,lease.rs}", "crates/scion-derived/tests/framework_*.rs" ], "depends_on": [ "P01", "P03" ], "deliverables": "Implement the P14 part of PLAN 2.7. Registry: register and look up by name and version; v1_default registers the commit_graph and path_index descriptors (computed by History), history and git_mapping as eager types (constructing scion_derived::history::HistoryType and git_mapping::GitMappingType, whose behaviour is implemented by P21), and blame as lazy (blame::BlameType, P24). latest_version; content_bearing_types. StoreDeriveCtx implements DeriveCtx over ObjectRepo: blob_or_empty returns empty for quarantined or purged blobs; meta returns None when redacted; put_object uses ObjectRepo.put_encoded. DerivedEngine: - impl EagerHook runs every registered eager type and concatenates their rows. required_eager lists eager (name, version) pairs for fsck. - get(ty, version, key) returns the stored value or derives lazily. The lazy algorithm uses an explicit work stack (NEVER recursion): find the nearest keys whose values are stored by walking deps(), then derive forward in dependency order, writing back every intermediate value with put_derived (ON CONFLICT DO NOTHING). - Leases (try_lease/release_lease) only deduplicate concurrent work. A derivation proceeds even if the lease is not acquired, after a short wait-and-recheck. - Version selection: None means the latest registered version; old versions remain readable. - get_stored. purge_content_for deletes records of content-bearing types whose source_object is the id. - Reports derive duration and backlog through the DeriveMetrics hook set via set_metrics (no scion-obs dependency). Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "19.6: two workers deriving the same key concurrently produce byte-identical values and exactly one row (test-only lazy type). A lazy derivation over a chain of 10^5 dependencies (default 10^4; #[ignore] full_ 10^5) completes on a thread with a 256 KiB stack (no recursion growth) and writes back every intermediate value. A version bump re-derives on query while the old version's values stay readable. A test-only type registered at runtime is queryable through DerivedEngine::get (the generic path) with no other change. Lease loss does not affect correctness. purge_content_for deletes only content-bearing records keyed by the id. The EagerHook output contains the rows of all registered test eager types.", "size": "M" }, { "id": "P15", "title": "Gated real backends: scion-store-crdb (CockroachDB) and scion-store-s3 (S3/MinIO)", "wave": 2, "spec_sections": "3.5, 5.1, 5.2, 5.3, 5.4, 16.1 classes 1/2, 19.3 (on real backends)", "owns": [ "crates/scion-store-crdb/src/**", "crates/scion-store-crdb/tests/**", "crates/scion-store-s3/src/**", "crates/scion-store-s3/tests/**" ], "depends_on": [ "P01", "P03" ], "deliverables": "Implement PLAN 2.4 following docs/contracts/storage-semantics.md exactly. CRDB: embedded MIGRATIONS create every SPEC 5.2 table keyed by repo_id first, plus audit_log and the secondary indexes listed in the contract; migrate is idempotent. Each MetaStore method runs in one SERIALIZABLE transaction via deadpool-postgres and tokio-postgres, with a client-side retry loop on SQLSTATE 40001 (bounded, with jitter; when exhausted, return UNAVAILABLE/txn_retry). Inserts use ON CONFLICT DO NOTHING. cas_ref runs UPDATE ... WHERE name=$1 AND version=$2 (or INSERT for Absent and DELETE for delete), inserts the ref_log, inserts landed rows (a unique violation 23505 rolls back and returns LandedConflict with the existing row) and inserts the audit row, all in the same transaction. Consistency::Stale uses AS OF SYSTEM TIME follower_read_timestamp(). changesets_by_prefix uses a range scan on the id column. Connection errors map to UNAVAILABLE/store_unavailable. connect_fresh_for_tests creates a unique database. S3: object_store AmazonS3 with endpoint, path-style and allow_http for MinIO; get on a missing key returns None; copy and delete are idempotent; list uses prefix listing. Keys are prefixed with key_prefix (unique per test run). Every test is gated: when SCION_TEST_CRDB_URL or SCION_TEST_S3_ENDPOINT is unset it prints 'SKIP: ...' and passes. Test names start with external_. Definition of done: cargo fmt --check, cargo build --workspace --all-targets, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and scripts/check-layers.sh all pass (scripts/check.sh), only owned files are touched, and every frozen signature in docs/PLAN.md section 2 is unchanged. Additive pub items are allowed only inside owned files; any other contract change goes into the final report as an amendment request.", "tests": "When the env vars are set: scion_store::conformance::meta_store_suite against CrdbMetaStore and blob_store_suite against S3BlobStore. Also: migration idempotency; 40001 retry exercised by two conflicting concurrent transactions; Stale reads may miss a just-committed row while Strong reads see it; 1000-way concurrent cas_ref yields exactly one winner. 19.3 on real backends: ObjectRepo over CRDB plus S3 with InstrumentedBlob faults (put fails) leaves no objects row; FailAfter on meta.insert_object leaves a durable row and bytes. When unset, every test prints SKIP and passes.", "size": "L" }, { "id": "P16", "title": "scion-daemon: Unix-socket caching tier, local authorization, redaction tracking, verbatim forwarding", "wave": 2, "spec_sections": "3.3, 6.3, 7.3 (daemon semantics), 7.8, 10.1-10.5, 13.3 (daemon), 16.1 classes 1/6/7, 17 (socket permissions), 18.8 (daemon loop), 19.8, 19.9, 19.11 (daemon), 19.12 (daemon), 19.13 (daemon)", "owns": [ "crates/scion-daemon/src/**", "crates/scion-daemon/tests/**" ], "depends_on": [ "P01", "P04", "P06", "P08", "P10" ],