Sync from development: forward-only paint branches, bounded hook walk, demo starts unpaused, lease-guarded mirror (#8)

Syncs the public tree with the development repository's main (62851b55), rebuilt with scripts/public-release/build-public-tree.py. ## Changes - place: branches under refs/heads/paint/ only move forward for Git clients; deletes and history rewrites are refused against the server's stored tip. - place: the tile hook walks a short prefix (64 commits) first when a push is built on main from before a history reset, and the walk has a hard budget with cached ref version lookups. - Public demo: the canvas mirror pushes with --force-with-lease, so another writer is reported, never overwritten; it pushes as soon as place answers after a start. - Public demo: place starts with its kill switch released on every start; use place_running = false for a lasting stop. - Tests: the access-log test runs in its own binary; new mutation checks P27 to P29. ## Checks - Tree is identical (diff -r) to a fresh build-public-tree.py run. - gitleaks dir: only the 4 known allowed hits (Git release-key fingerprint, mock operator token, Azure Terraform placeholders, JWT fuzz corpus). - Grep for internal links, internal names, secret-manager references, non-placeholder AWS account IDs and internal hostnames: no hits.

reason-machines[bot] committed 1 day agomain

Commit 02197261 parent 337d6a616 files · +960 −226Browse files

Unified · Split

16 files

.github/workflows/ci.yml +2 −0
crates/aspen-server/tests/access_log.rs +165 −0
crates/aspen-server/tests/observability.rs +0 −123
demo/place/docs/TILES.md +7 −3
demo/place/server/scripts/check-mutations.py +9 −1
demo/place/server/src/hook.rs +378 −55
demo/place/server/src/rules.rs +12 −0
demo/place/server/tests/place.rs +146 −5
deploy/terraform/aws/public-demo/README.md +17 −10
deploy/terraform/aws/public-demo/environment/canvas-mirror.sh +53 −8
deploy/terraform/aws/public-demo/environment/demo.tfvars.example +0 −4
deploy/terraform/aws/public-demo/environment/place.tf +0 −1
deploy/terraform/aws/public-demo/environment/tests/canvas-mirror.test.sh +150 −0
deploy/terraform/aws/public-demo/environment/tests/environment.tftest.hcl +8 −0
deploy/terraform/aws/public-demo/environment/variables.tf +0 −6
docs/PLAN.md +13 −10

.github/workflows/ci.yml+2−0

@@ -190,6 +190,8 @@ jobs: · 189 unmodified lines
190	190	PG_BIN="$PG_BIN" sh aws/public-demo/environment/tests/create-database.test.sh
191	191	- name: public demo smoke.sh keeps the token off git's command line
192	192	run: sh aws/public-demo/tests/smoke.test.sh
	193	+ - name: public demo canvas-mirror.sh pushes as soon as place answers after a start
	194	+ run: sh aws/public-demo/environment/tests/canvas-mirror.test.sh
193	195	- name: tflint
194	196	env:
195	197	GITHUB_TOKEN: ${{ github.token }} # plugin download rate limit

crates/aspen-server/tests/access_log.rs+165−0

@@ -0,0 +1,165 @@
	1	+//! The access log of the Git router (node C5), as a log reader sees it.
	2	+//!
	3	+//! The log is captured by a global `tracing` subscriber, so this file holds
	4	+//! this one test and nothing else. A thread-local subscriber
	5	+//! (`tracing::subscriber::set_default`) is not enough in a binary whose
	6	+//! other tests also serve Git requests: while it is the only registered
	7	+//! dispatcher, `tracing-core` computes the interest of a callsite first hit
	8	+//! on another thread from that thread's default (no subscriber) and caches
	9	+//! it as never, so the `git request` event was dropped on every thread for
	10	+//! the rest of the process (issue #97). A global subscriber is every
	11	+//! thread's default, so no thread can register a callsite against nothing.
	12	+
	13	+use std::sync::Arc;
	14	+
	15	+use aspen_auth::{AuthService, Limits, LimitsConfig, MemoryTokenStore, PolicyConfig};
	16	+use aspen_blob::LocalBlobStore;
	17	+use aspen_core::meta::MetaStore;
	18	+use aspen_meta::SqlStore;
	19	+use aspen_pack::MaxObjectBytes;
	20	+use aspen_server::access::{Access, ClientAddr};
	21	+use aspen_server::{Server, ServerConfig};
	22	+use axum::body::Body;
	23	+use axum::http::Request;
	24	+use tower::ServiceExt as _;
	25	+
	26	+/// A SQLite store in `dir`, migrated, with a pool of at most
	27	+/// `connections`.
	28	+async fn sqlite_store(dir: &std::path::Path, connections: u32) -> SqlStore {
	29	+ let url = format!("sqlite://{}?mode=rwc", dir.join("meta.db").display());
	30	+ let opts = aspen_meta::StoreOptions {
	31	+ max_connections: connections,
	32	+ ..aspen_meta::StoreOptions::default()
	33	+ };
	34	+ let meta = SqlStore::connect_with(&url, &opts).await.unwrap();
	35	+ meta.migrate().await.unwrap();
	36	+ meta
	37	+}
	38	+
	39	+fn server_config() -> ServerConfig {
	40	+ ServerConfig::new(MaxObjectBytes::new(1 << 20).unwrap())
	41	+}
	42	+
	43	+/// A `MakeWriter` collecting formatted log lines.
	44	+#[derive(Clone, Default)]
	45	+struct Captured(Arc<std::sync::Mutex<Vec<u8>>>);
	46	+
	47	+impl std::io::Write for Captured {
	48	+ fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
	49	+ self.0.lock().unwrap().extend_from_slice(buf);
	50	+ Ok(buf.len())
	51	+ }
	52	+ fn flush(&mut self) -> std::io::Result<()> {
	53	+ Ok(())
	54	+ }
	55	+}
	56	+
	57	+impl Captured {
	58	+ fn lines(&self) -> Vec<String> {
	59	+ String::from_utf8(self.0.lock().unwrap().clone())
	60	+ .unwrap()
	61	+ .lines()
	62	+ .map(str::to_owned)
	63	+ .collect()
	64	+ }
	65	+}
	66	+
	67	+/// The access log covers requests the guard refuses as well as those it
	68	+/// admits: one `git request` event each, with the status, in a
	69	+/// `git_request` span holding the method and the path. An admitted request
	70	+/// also names the repository and endpoint the guard resolved. Neither the
	71	+/// query string nor the credential is ever logged.
	72	+#[tokio::test]
	73	+async fn the_access_log_covers_refused_requests_without_credentials_or_queries() {
	74	+ let dir = tempfile::tempdir().unwrap();
	75	+ let meta = sqlite_store(dir.path(), 10).await;
	76	+ meta.create_repo(
	77	+ &aspen_core::name::RepoName::new("acme/obs").unwrap(),
	78	+ &aspen_core::repo::RepoSettings::new(
	79	+ aspen_core::name::RefName::new("refs/heads/main").unwrap(),
	80	+ )
	81	+ .unwrap(),
	82	+ )
	83	+ .await
	84	+ .unwrap();
	85	+ let blobs = LocalBlobStore::open(dir.path().join("blobs")).unwrap();
	86	+ let policy = PolicyConfig {
	87	+ anonymous: vec!["read:*".parse().unwrap()],
	88	+ ..PolicyConfig::default()
	89	+ };
	90	+ let access = Arc::new(Access::new(
	91	+ AuthService::new(policy, MemoryTokenStore::new()).unwrap(),
	92	+ Limits::new(LimitsConfig::default()),
	93	+ meta.clone(),
	94	+ ClientAddr::Peer,
	95	+ ));
	96	+ let router = Server::new(meta, blobs, server_config()).router(access);
	97	+
	98	+ let captured = Captured::default();
	99	+ let writer = captured.clone();
	100	+ let subscriber = tracing_subscriber::fmt()
	101	+ .with_writer(move \|\| writer.clone())
	102	+ .with_ansi(false)
	103	+ .with_max_level(tracing::Level::INFO)
	104	+ .finish();
	105	+ // Global, not `set_default`: see the module comment.
	106	+ tracing::subscriber::set_global_default(subscriber).unwrap();
	107	+
	108	+ // "x:not-a-token": a credential that is presented but invalid.
	109	+ const CREDENTIAL: &str = "eDpub3QtYS10b2tlbg==";
	110	+ let requests = [
	111	+ // Admitted: anonymous may read.
	112	+ ("/acme/obs.git/info/refs?service=git-upload-pack", None, 200),
	113	+ // An invalid credential is refused, never downgraded.
	114	+ (
	115	+ "/acme/obs.git/info/refs?service=git-upload-pack",
	116	+ Some(CREDENTIAL),
	117	+ 401,
	118	+ ),
	119	+ // Pushing needs write, even for the advertisement.
	120	+ (
	121	+ "/acme/obs.git/info/refs?service=git-receive-pack",
	122	+ None,
	123	+ 401,
	124	+ ),
	125	+ // Not a Git path.
	126	+ ("/favicon.ico", None, 404),
	127	+ ];
	128	+ for (uri, credential, status) in requests {
	129	+ let mut request = Request::get(uri);
	130	+ if let Some(c) = credential {
	131	+ request = request.header("authorization", format!("Basic {c}"));
	132	+ }
	133	+ let response = router
	134	+ .clone()
	135	+ .oneshot(request.body(Body::empty()).unwrap())
	136	+ .await
	137	+ .unwrap();
	138	+ assert_eq!(response.status().as_u16(), status, "{uri}");
	139	+ }
	140	+
	141	+ let lines: Vec<String> = captured
	142	+ .lines()
	143	+ .into_iter()
	144	+ .filter(\|l\| l.contains("git request"))
	145	+ .collect();
	146	+ assert_eq!(lines.len(), requests.len(), "{lines:#?}");
	147	+ for ((uri, _, status), line) in requests.iter().zip(&lines) {
	148	+ let path = uri.split('?').next().unwrap();
	149	+ assert!(line.contains(&format!("status={status}")), "{line}");
	150	+ assert!(line.contains(&format!("path=\"{path}\"")), "{line}");
	151	+ assert!(line.contains("method=GET"), "{line}");
	152	+ assert!(!line.contains("service="), "query logged: {line}");
	153	+ assert!(!line.contains(CREDENTIAL), "credential logged: {line}");
	154	+ assert!(!line.contains("not-a-token"), "credential logged: {line}");
	155	+ }
	156	+ assert!(lines[0].contains("repo=acme/obs"), "{}", lines[0]);
	157	+ assert!(
	158	+ lines[0].contains("endpoint=\"upload_pack_advertisement\""),
	159	+ "{}",
	160	+ lines[0]
	161	+ );
	162	+ for line in &lines[1..] {
	163	+ assert!(!line.contains("repo="), "refused before routing: {line}");
	164	+ }
	165	+}

crates/aspen-server/tests/observability.rs+0−123

@@ -817,129 +817,6 @@ async fn a_hung_database_read_times_out_and_serves_the_last_values() { · 816 unmodified lines
817	817	assert!(after.value("aspen_meta_stats_timestamp_seconds", &[]) > read_at);
818	818	}
819	819
820		−/// A `MakeWriter` collecting formatted log lines.
821		−#[derive(Clone, Default)]
822		−struct Captured(Arc<std::sync::Mutex<Vec<u8>>>);
823		−
824		−impl std::io::Write for Captured {
825		− fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
826		− self.0.lock().unwrap().extend_from_slice(buf);
827		− Ok(buf.len())
828		− }
829		− fn flush(&mut self) -> std::io::Result<()> {
830		− Ok(())
831		− }
832		−}
833		−
834		−impl Captured {
835		− fn lines(&self) -> Vec<String> {
836		− String::from_utf8(self.0.lock().unwrap().clone())
837		− .unwrap()
838		− .lines()
839		− .map(str::to_owned)
840		− .collect()
841		− }
842		−}
843		−
844		−/// The access log covers requests the guard refuses as well as those it
845		−/// admits: one `git request` event each, with the status, in a
846		−/// `git_request` span holding the method and the path. An admitted request
847		−/// also names the repository and endpoint the guard resolved. Neither the
848		−/// query string nor the credential is ever logged.
849		−#[tokio::test]
850		−async fn the_access_log_covers_refused_requests_without_credentials_or_queries() {
851		− let dir = tempfile::tempdir().unwrap();
852		− let meta = sqlite_store(dir.path(), 10).await;
853		− meta.create_repo(
854		− &aspen_core::name::RepoName::new("acme/obs").unwrap(),
855		− &aspen_core::repo::RepoSettings::new(
856		− aspen_core::name::RefName::new("refs/heads/main").unwrap(),
857		− )
858		− .unwrap(),
859		− )
860		− .await
861		− .unwrap();
862		− let blobs = LocalBlobStore::open(dir.path().join("blobs")).unwrap();
863		− let policy = PolicyConfig {
864		− anonymous: vec!["read:*".parse().unwrap()],
865		− ..PolicyConfig::default()
866		− };
867		− let access = Arc::new(Access::new(
868		− AuthService::new(policy, MemoryTokenStore::new()).unwrap(),
869		− Limits::new(LimitsConfig::default()),
870		− meta.clone(),
871		− ClientAddr::Peer,
872		− ));
873		− let router = Server::new(meta, blobs, server_config()).router(access);
874		−
875		− let captured = Captured::default();
876		− let writer = captured.clone();
877		− let subscriber = tracing_subscriber::fmt()
878		− .with_writer(move \|\| writer.clone())
879		− .with_ansi(false)
880		− .with_max_level(tracing::Level::INFO)
881		− .finish();
882		− let _guard = tracing::subscriber::set_default(subscriber);
883		−
884		− // "x:not-a-token": a credential that is presented but invalid.
885		− const CREDENTIAL: &str = "eDpub3QtYS10b2tlbg==";
886		− let requests = [
887		− // Admitted: anonymous may read.
888		− ("/acme/obs.git/info/refs?service=git-upload-pack", None, 200),
889		− // An invalid credential is refused, never downgraded.
890		− (
891		− "/acme/obs.git/info/refs?service=git-upload-pack",
892		− Some(CREDENTIAL),
893		− 401,
894		− ),
895		− // Pushing needs write, even for the advertisement.
896		− (
897		− "/acme/obs.git/info/refs?service=git-receive-pack",
898		− None,
899		− 401,
900		− ),
901		− // Not a Git path.
902		− ("/favicon.ico", None, 404),
903		− ];
904		− for (uri, credential, status) in requests {
905		− let mut request = Request::get(uri);
906		− if let Some(c) = credential {
907		− request = request.header("authorization", format!("Basic {c}"));
908		− }
909		− let response = router
910		− .clone()
911		− .oneshot(request.body(Body::empty()).unwrap())
912		− .await
913		− .unwrap();
914		− assert_eq!(response.status().as_u16(), status, "{uri}");
915		− }
916		−
917		− let lines: Vec<String> = captured
918		− .lines()
919		− .into_iter()
920		− .filter(\|l\| l.contains("git request"))
921		− .collect();
922		− assert_eq!(lines.len(), requests.len(), "{lines:#?}");
923		− for ((uri, _, status), line) in requests.iter().zip(&lines) {
924		− let path = uri.split('?').next().unwrap();
925		− assert!(line.contains(&format!("status={status}")), "{line}");
926		− assert!(line.contains(&format!("path=\"{path}\"")), "{line}");
927		− assert!(line.contains("method=GET"), "{line}");
928		− assert!(!line.contains("service="), "query logged: {line}");
929		− assert!(!line.contains(CREDENTIAL), "credential logged: {line}");
930		− assert!(!line.contains("not-a-token"), "credential logged: {line}");
931		− }
932		− assert!(lines[0].contains("repo=acme/obs"), "{}", lines[0]);
933		− assert!(
934		− lines[0].contains("endpoint=\"upload_pack_advertisement\""),
935		− "{}",
936		− lines[0]
937		− );
938		− for line in &lines[1..] {
939		− assert!(!line.contains("repo="), "refused before routing: {line}");
940		− }
941		−}
942		−
943	820	fn repo_file(rel: &str) -> String {
944	821	let p = Path::new(env!("CARGO_MANIFEST_DIR"))
945	822	.join("../..")

demo/place/docs/TILES.md+7−3

@@ -88,8 +88,11 @@ Worked example, with the default cap of 64 tiles: · 87 unmodified lines
88	88	painting 30 more tiles, and pushes. Her client sends a zero old-oid. The
89	89	range is `A2 ^T1 ^A1` = {`A2`}: 30 tiles, accepted. Counting from the
90	90	zero old-oid (`A2 ^T1`) would include `A1` and refuse 70 tiles.
91		−3. Ada rewrites her branch as `B1` on `T0` (60 tiles) and force-pushes. The
92		− same rule applies: `B1 ^T1 ^A2` = {`B1`}: 60 tiles, accepted.
	91	+3. Ada rewrites her branch as `B1` on `T0` (60 tiles) and force-pushes.
	92	+ `paint/` branches only move forward, so that push is refused; she
	93	+ pushes `B1` to a new branch, `refs/heads/paint/ada-2`: `B1 ^T1` =
	94	+ {`B1`}: 60 tiles, accepted. On a branch outside `paint/`, which Aspen
	95	+ lets a client force, the same rule gives `B1 ^T1 ^A2` = {`B1`}.
93	96
94	97	A rename of a tile file shows up as a delete plus an add, so it is refused
95	98	with `delete-not-allowed`.
@@ -101,7 +104,8 @@ with `delete-not-allowed`. · 5 unmodified lines
101	104	\| Only branches (`refs/heads/*`) can be pushed; tags and other refs are refused. Branch names are at most 100 characters after `refs/heads/`. \| `bad-ref` \|
102	105	\| `refs/heads/main` belongs to the merger bot. Push to any other branch. \| `protected-ref` \|
103	106	\| Branches under `refs/heads/visitors/`, `refs/heads/agent/` and `refs/heads/swarm/` belong to the web page's visitors and the server's agents; Git clients may not push to them. \| `protected-ref` \|
104		−\| Deleting a branch (new tip all zeros) is allowed; no commit rules apply. Whether a given client may delete or force-push a branch is Aspen's ref-update policy (spike S2), not a tile rule. \| (none) \|
	107	+\| Branches under `refs/heads/paint/` only move forward: a Git client may create one and fast-forward it, but not delete it or rewrite its history (a force-push, `-o force` or a rewind), because anyone can push to any branch and it may be someone else's. Push the rewrite to a new branch instead. The check compares the stored tip with the new one, never the client's old value. \| (none; the `ng` line says so) \|
	108	+\| Deleting any other branch (new tip all zeros) is allowed; no commit rules apply. Whether a given client may delete or force-push such a branch is Aspen's ref-update policy (spike S2), not a tile rule. \| (none) \|
105	109	\| Every changed path is under `tiles/`. \| `outside-tiles` \|
106	110	\| Every path under `tiles/` is a valid tile name. \| `bad-tile-path` \|
107	111	\| Every tile is inside the canvas. \| `out-of-bounds` \|

demo/place/server/scripts/check-mutations.py+9−1

@@ -105,8 +105,16 @@ muts = [ · 104 unmodified lines
105	105	HIST + ["clones_and_branches_from_before_a_reset"]),
106	106	("P25 the merger never starts main's history again", [(PLACE, " if max == 0 \|\| self.with_state(\|s\| s.main.history.bytes) < max {", " if true \|\| self.with_state(\|s\| s.main.history.bytes) < max {")],
107	107	HIST + ["a_full_clone_stays_within_the_history_limit"]),
108		− ("P26 a restart forgets the history's size", [(PLACE, " }, \|h\| h.grow(0)),", " }, \|_\| History::root(tip, 0)),")],
	108	+ ("P26 a restart forgets the history's size", [(PLACE, " \|h\| h.grow(0),\n", " \|_\| History::root(tip, 0),\n")],
109	109	HIST + ["clones_and_branches_from_before_a_reset"]),
	110	+ # The hook's walk of a push built on an earlier main stays bounded.
	111	+ ("P27 unreadable IDs are not charged to the boundary walk", [(HOOK, " spent += 1;\n match objects.read(id).await? {", " match objects.read(id).await? {")],
	112	+ ["-p", "aspen-place", "--lib", "the_boundary_walk_stays_within_its_budget_on_a_wide_generation"]),
	113	+ ("P28 the boundary walk asks about a whole generation at once", [(HOOK, "let n = queue.len().min(budget - spent).min(CLASSIFY_BATCH);", "let n = queue.len();"),
	114	+ (HOOK, " if spent + queue.len() >= budget {\n break;\n }\n", "")],
	115	+ ["-p", "aspen-place", "--lib", "the_boundary_walk_stays_within_its_budget_on_a_wide_generation"]),
	116	+ ("P29 the hook walks the whole range before looking for an earlier main", [(HOOK, "let probe = PROBE_COMMITS.min(max_commits);", "let probe = max_commits;")],
	117	+ ["-p", "aspen-place", "--lib", "a_push_on_an_earlier_main_costs_the_same_whatever_its_length"]),
110	118	]
111	119
112	120	only = sys.argv[1:]

demo/place/server/src/hook.rs+378−55

@@ -4,15 +4,18 @@ · 3 unmodified lines
4	4	//! For a Git client's update of a branch the hook:
5	5	//!
6	6	//! 1. checks the ref (`main` and the server's own namespaces are
7		−//! protected, only branches are accepted);
	7	+//! protected, only branches are accepted), and refuses a delete or a
	8	+//! non-fast-forward of a `paint/` branch ([`rules::forward_only`]),
	9	+//! judged against the stored tip;
8	10	//! 2. takes the commits `git rev-list <new> ^<trunk tip> ^<stored tip>`
9	11	//! lists, where both excluded tips are the ones the server read
10	12	//! ([`HookUpdate::trunk`], [`HookUpdate::stored`]), never the client's
11		−//! old value (A9 review obligation 4). When that range runs into a
12		−//! root commit or past the commit limit, the push may be built on a
13		−//! `main` from before a history reset ([`crate::place::Place::reset_history`]):
14		−//! the commits of any earlier `main` it reaches are excluded too
15		−//! ([`MainHistory`]), so a clone made before a reset still pushes;
	13	+//! old value (A9 review obligation 4). When that range, walked at most
	14	+//! [`PROBE_COMMITS`] deep, runs into a root commit or past that depth,
	15	+//! the push may be built on a `main` from before a history reset
	16	+//! ([`crate::place::Place::reset_history`]): the commits of any earlier
	17	+//! `main` it reaches are excluded too ([`MainHistory`]), so a clone made
	18	+//! before a reset still pushes, without walking the old history first;
16	19	//! 3. diffs each commit against its first parent and runs
17	20	//! [`rules::validate_push`] over them;
18	21	//! 4. charges the push to the caller's per-branch quota.
@@ -23,7 +26,7 @@ · 4 unmodified lines
23	26	//! (web strokes, agents, the merger bot) carry [`INTERNAL_KEY`] and were
24	27	//! checked before they were written.
25	28
26		−use std::collections::HashMap;
	29	+use std::collections::{HashMap, HashSet, VecDeque};
27	30	use std::sync::{Arc, Mutex};
28	31
29	32	use aspen_core::error::StoreError;
@@ -225,13 +228,21 @@ pub enum Indexed { · 195 unmodified lines
225	228	/// checked: a push built on it is checked from there, like one built on
226	229	/// the current `main`.
227	230	pub trait MainHistory: Send + Sync + std::fmt::Debug {
228		− /// One answer per ID, in order.
	231	+ /// One answer per ID, in order. `versions` is what earlier calls of
	232	+ /// the same walk learned about ref versions; it lives as long as the
	233	+ /// walk, so each version is looked up at most once per walk.
229	234	fn classify<'a>(
230	235	&'a self,
231	236	ids: &'a [ObjectId],
	237	+ versions: &'a mut VersionCache,
232	238	) -> HookFuture<'a, Result<Vec<Indexed>, StoreError>>;
233	239	}
234	240
	241	+/// Whether each ref version a walk met is one of `main`'s, kept for the
	242	+/// whole walk ([`MainHistory::classify`]).
	243	+#[derive(Debug, Default)]
	244	+pub struct VersionCache(HashMap<RefVersionId, bool>);
	245	+
235	246	/// [`MainHistory`] from Aspen's commit index: a commit's row names a
236	247	/// version whose tip reaches it, and a commit is on `main`'s history
237	248	/// exactly when that version is one of `main`'s (`main` takes no client
@@ -249,13 +260,14 @@ impl<M: MetaStore + std::fmt::Debug + 'static> MainHistory for IndexedMain<M> { · 11 unmodified lines
249	260	fn classify<'a>(
250	261	&'a self,
251	262	ids: &'a [ObjectId],
	263	+ versions: &'a mut VersionCache,
252	264	) -> HookFuture<'a, Result<Vec<Indexed>, StoreError>> {
253	265	Box::pin(async move {
254	266	let sorted = SortedIds::from_unsorted(ids.to_vec());
255	267	let rows = self.meta.lookup_commits(self.repo, &sorted).await?;
256	268	let by_id: HashMap<ObjectId, CommitLookup> =
257	269	sorted.as_slice().iter().copied().zip(rows).collect();
258		− let mut on_main: HashMap<RefVersionId, bool> = HashMap::new();
	270	+ let on_main = &mut versions.0;
259	271	let mut out = Vec::with_capacity(ids.len());
260	272	for id in ids {
261	273	let Some(CommitLookup::Indexed(row)) = by_id.get(id) else {
@@ -409,31 +421,52 @@ async fn check_git_update( · 147 unmodified lines
409	421	if let Some(r) = rules::validate_ref(name, Pusher::Git) {
410	422	return Ok(refuse(vec![r]));
411	423	}
	424	+ let forward_only = rules::forward_only(name);
412	425	let Some(new) = update.new else {
413		− // A delete passed `validate_ref`, so the branch is not reserved;
414		− // any Git client may delete any such branch (docs/TILES.md: who may
415		− // delete is Aspen's ref-update policy, not a tile rule). No commit
416		− // rules apply.
	426	+ if forward_only {
	427	+ return Ok(refuse_one(format!(
	428	+ "{name}: {prefix} branches cannot be deleted, because anyone can push to any of them; leave it (it expires once its tiles merge) and push to a new branch, e.g. git push origin HEAD:refs/heads/{prefix}<you>-2",
	429	+ prefix = rules::FORWARD_ONLY_PREFIX
	430	+ )));
	431	+ }
	432	+ // A delete of any other unreserved branch is Aspen's ref-update
	433	+ // policy, not a tile rule. No commit rules apply.
417	434	return Ok(HookVerdict::Accept);
418	435	};
419		− let mut exclude: Vec<ObjectId> = update.trunk.into_iter().chain(update.stored).collect();
420		− let limits = shared.limits;
421		− let mut range = objects.range(new, &exclude, limits.max_commits).await?;
422		− let reaches_a_root = match &range {
423		− CommitRange::Commits(c) => c.iter().any(\|c\| c.parents.is_empty()),
424		− CommitRange::TooLong => true,
425		− CommitRange::Undecided => false,
426		− };
427		− if reaches_a_root {
428		− // Built on a `main` from before a history reset (or on no `main` at
429		− // all): stop at the earlier `main`'s commits too. Ordinary pushes
430		− // never get here, so they pay nothing for it.
431		− let boundary = main_boundary(shared, objects, new, limits.max_commits).await?;
432		− if !boundary.is_empty() {
433		− exclude.extend(boundary);
434		− range = objects.range(new, &exclude, limits.max_commits).await?;
	436	+ if forward_only
	437	+ && let Some(stored) = update.stored
	438	+ && stored != new
	439	+ {
	440	+ // Fast-forward exactly when the stored tip is an ancestor of the
	441	+ // new one: the walk from `stored` that excludes `new` then lists
	442	+ // nothing, and with a limit of 0 any commit it would list is
	443	+ // `TooLong`. The stored tip is the server's, never the client's
	444	+ // old value, so `-o force` or a stale lease cannot get past it.
	445	+ match objects.range(stored, &[new], 0).await? {
	446	+ CommitRange::Commits(_) => {}
	447	+ CommitRange::TooLong => {
	448	+ return Ok(refuse_one(format!(
	449	+ "{name}: {prefix} branches only move forward, so a force-push cannot rewrite one (it may be someone else's); push your history to a new branch instead, e.g. git push origin HEAD:refs/heads/{prefix}<you>-2",
	450	+ prefix = rules::FORWARD_ONLY_PREFIX
	451	+ )));
	452	+ }
	453	+ CommitRange::Undecided => {
	454	+ return Ok(refuse_one(
	455	+ "the server could not read this push's history yet; retry".to_owned(),
	456	+ ));
	457	+ }
435	458	}
436	459	}
	460	+ let exclude: Vec<ObjectId> = update.trunk.into_iter().chain(update.stored).collect();
	461	+ let limits = shared.limits;
	462	+ let range = commit_range(
	463	+ &*shared.main_history,
	464	+ objects,
	465	+ new,
	466	+ exclude,
	467	+ limits.max_commits,
	468	+ )
	469	+ .await?;
437	470	let range = match range {
438	471	CommitRange::Commits(c) => c,
439	472	CommitRange::TooLong => {
@@ -506,47 +539,114 @@ async fn check_git_update( · 66 unmodified lines
506	539	Ok(HookVerdict::Accept)
507	540	}
508	541
	542	+/// How deep the hook walks a push's range before it also looks for an
	543	+/// earlier `main`'s commits. A range this short that reaches no root
	544	+/// commit is an ordinary push, checked with one walk as before; a push
	545	+/// built on a `main` from before a history reset reads at most this many
	546	+/// commits of the old history before the hook stops at it.
	547	+pub const PROBE_COMMITS: usize = 64;
	548	+
	549	+/// Most IDs [`main_boundary`] asks [`MainHistory::classify`] about at once.
	550	+const CLASSIFY_BATCH: usize = 512;
	551	+
	552	+/// The commits `new` adds over `exclude`, at most `max_commits` of them
	553	+/// ([`HookObjects::range`]), stopping at the commits of any version of
	554	+/// `main` too, past or present ([`main_boundary`]). The range is first
	555	+/// walked [`PROBE_COMMITS`] deep; only a push that reaches a root commit
	556	+/// or goes deeper pays for the boundary walk and a second range walk.
	557	+async fn commit_range(
	558	+ history: &dyn MainHistory,
	559	+ objects: &mut dyn HookObjects,
	560	+ new: ObjectId,
	561	+ mut exclude: Vec<ObjectId>,
	562	+ max_commits: usize,
	563	+) -> Result<CommitRange, StoreError> {
	564	+ let probe = PROBE_COMMITS.min(max_commits);
	565	+ let first = objects.range(new, &exclude, probe).await?;
	566	+ let reaches_a_root = match &first {
	567	+ CommitRange::Commits(c) => c.iter().any(\|c\| c.parents.is_empty()),
	568	+ CommitRange::TooLong => true,
	569	+ CommitRange::Undecided => false,
	570	+ };
	571	+ if !reaches_a_root {
	572	+ return Ok(first);
	573	+ }
	574	+ // Built on a `main` from before a history reset, on no `main` at all,
	575	+ // or simply long: stop at the earlier `main`'s commits too, before
	576	+ // walking any further.
	577	+ let boundary = main_boundary(history, objects, new, max_commits).await?;
	578	+ if boundary.is_empty() && (matches!(first, CommitRange::Commits(_)) \|\| probe == max_commits) {
	579	+ // No `main` to stop at, and the first walk already saw the whole
	580	+ // range (an unrelated history) or went as deep as allowed.
	581	+ return Ok(first);
	582	+ }
	583	+ exclude.extend(boundary);
	584	+ objects.range(new, &exclude, max_commits).await
	585	+}
	586	+
509	587	/// The commits of `main`'s history (any epoch, [`MainHistory`]) that the
510		−/// history of `new` reaches first: walking down from `new`, a commit on
511		−/// `main` is kept and not walked past. Gives up (returns what it found so
512		−/// far) after `max_commits` commits that are not on `main`, which the
513		−/// range then reports as too long. Commits the index knows are read from
514		−/// it, in one batch per generation; the push's own commits are read from
515		−/// its pack.
	588	+/// history of `new` reaches first: walking down from `new` breadth first,
	589	+/// a commit on `main` is kept and not walked past.
	590	+///
	591	+/// The walk spends at most `budget` units in all, one per ID it asks the
	592	+/// index about and one per object it reads, whatever the shape of the
	593	+/// history (a wide octopus generation, a deep chain off `main`, IDs that
	594	+/// cannot be read). Once the budget is spent it gives up and returns what
	595	+/// it found so far; every ID it returns is on `main`, so a partial answer
	596	+/// only leaves the range longer, which then reports it as too long.
	597	+/// Commits the index knows are classified from it in batches; the push's
	598	+/// own commits are read from its pack.
516	599	async fn main_boundary(
517		− shared: &Shared,
	600	+ history: &dyn MainHistory,
518	601	objects: &mut dyn HookObjects,
519	602	new: ObjectId,
520		− max_commits: usize,
	603	+ budget: usize,
521	604	) -> Result<Vec<ObjectId>, StoreError> {
522	605	let mut boundary = Vec::new();
523		− let mut seen: std::collections::HashSet<ObjectId> = [new].into();
524		− let mut frontier = vec![new];
525		− let mut walked = 0usize;
526		− while !frontier.is_empty() && walked <= max_commits {
527		− let answers = shared.main_history.classify(&frontier).await?;
528		− let mut next = Vec::new();
529		− for (id, answer) in frontier.iter().zip(answers) {
	606	+ let mut seen: HashSet<ObjectId> = [new].into();
	607	+ let mut queue: VecDeque<ObjectId> = [new].into();
	608	+ let mut versions = VersionCache::default();
	609	+ let mut spent = 0usize;
	610	+ while !queue.is_empty() && spent < budget {
	611	+ let n = queue.len().min(budget - spent).min(CLASSIFY_BATCH);
	612	+ let batch: Vec<ObjectId> = queue.drain(..n).collect();
	613	+ spent += batch.len();
	614	+ let answers = history.classify(&batch, &mut versions).await?;
	615	+ for (id, answer) in batch.iter().zip(answers) {
530	616	let parents = match answer {
531	617	Indexed::Main => {
532	618	boundary.push(*id);
533	619	continue;
534	620	}
535	621	Indexed::Other(parents) => parents,
536		− Indexed::NotIndexed => match objects.read(*id).await? {
537		− Some((ObjectKind::Commit, data)) => {
538		− parse_commit(&data)
539		− .map_err(\|what\| StoreError::Corrupt { what })?
540		− .1
	622	+ Indexed::NotIndexed => {
	623	+ if spent >= budget {
	624	+ return Ok(boundary);
	625	+ }
	626	+ spent += 1;
	627	+ match objects.read(*id).await? {
	628	+ Some((ObjectKind::Commit, data)) => {
	629	+ parse_commit(&data)
	630	+ .map_err(\|what\| StoreError::Corrupt { what })?
	631	+ .1
	632	+ }
	633	+ // Not readable or not a commit: the range decides.
	634	+ _ => continue,
541	635	}
542		− // Not readable or not a commit: the range decides.
543		− _ => continue,
544		− },
	636	+ }
545	637	};
546		− walked += 1;
547		− next.extend(parents.into_iter().filter(\|p\| seen.insert(*p)));
	638	+ // Every queued ID costs at least one unit, so IDs past what the
	639	+ // budget can still pay for would never be looked at: do not keep
	640	+ // them (an octopus commit may name any number of parents).
	641	+ for p in parents {
	642	+ if spent + queue.len() >= budget {
	643	+ break;
	644	+ }
	645	+ if seen.insert(p) {
	646	+ queue.push_back(p);
	647	+ }
	648	+ }
548	649	}
549		− frontier = next;
550	650	}
551	651	Ok(boundary)
552	652	}
@@ -788,3 +888,226 @@ mod tests { · 235 unmodified lines
788	888	);
789	889	}
790	890	}
	891	+
	892	+/// What the walks of a push's history cost: every ID asked of the commit
	893	+/// index and every object read is counted, against an in-memory history.
	894	+#[cfg(test)]
	895	+mod walk_tests {
	896	+ use super::*;
	897	+ use aspen_server::receive::hook::RangeCommit;
	898	+ use std::sync::atomic::{AtomicUsize, Ordering};
	899	+
	900	+ fn oid(n: usize) -> ObjectId {
	901	+ crate::git::object_id(ObjectKind::Blob, format!("c{n}").as_bytes())
	902	+ }
	903	+
	904	+ /// The commit index: what it says about each commit, and how many IDs
	905	+ /// it was asked about.
	906	+ #[derive(Debug, Default)]
	907	+ struct Index {
	908	+ rows: HashMap<ObjectId, Indexed>,
	909	+ asked: AtomicUsize,
	910	+ largest_batch: AtomicUsize,
	911	+ }
	912	+
	913	+ impl MainHistory for Index {
	914	+ fn classify<'a>(
	915	+ &'a self,
	916	+ ids: &'a [ObjectId],
	917	+ _: &'a mut VersionCache,
	918	+ ) -> HookFuture<'a, Result<Vec<Indexed>, StoreError>> {
	919	+ self.asked.fetch_add(ids.len(), Ordering::Relaxed);
	920	+ self.largest_batch.fetch_max(ids.len(), Ordering::Relaxed);
	921	+ let out = ids
	922	+ .iter()
	923	+ .map(\|id\| self.rows.get(id).cloned().unwrap_or(Indexed::NotIndexed))
	924	+ .collect();
	925	+ Box::pin(async move { Ok(out) })
	926	+ }
	927	+ }
	928	+
	929	+ /// The objects a push can read: commits with their parents. `range`
	930	+ /// walks down from the new tip like receive-pack's, reading each commit
	931	+ /// it visits and stopping at excluded tips and their ancestors.
	932	+ #[derive(Debug, Default)]
	933	+ struct Objects {
	934	+ parents: HashMap<ObjectId, Vec<ObjectId>>,
	935	+ reads: usize,
	936	+ }
	937	+
	938	+ impl Objects {
	939	+ fn add(&mut self, id: ObjectId, parents: Vec<ObjectId>) {
	940	+ self.parents.insert(id, parents);
	941	+ }
	942	+ }
	943	+
	944	+ impl HookObjects for Objects {
	945	+ fn range<'a>(
	946	+ &'a mut self,
	947	+ new: ObjectId,
	948	+ exclude: &'a [ObjectId],
	949	+ max_commits: usize,
	950	+ ) -> HookFuture<'a, Result<CommitRange, StoreError>> {
	951	+ // Ancestors of the excluded tips (not charged: the real walk
	952	+ // tests ancestry against them without visiting below them).
	953	+ let mut hidden: HashSet<ObjectId> = HashSet::new();
	954	+ let mut stack: Vec<ObjectId> = exclude.to_vec();
	955	+ while let Some(id) = stack.pop() {
	956	+ if hidden.insert(id) {
	957	+ stack.extend(self.parents.get(&id).into_iter().flatten().copied());
	958	+ }
	959	+ }
	960	+ let mut out = Vec::new();
	961	+ let mut seen: HashSet<ObjectId> = [new].into();
	962	+ let mut queue: VecDeque<ObjectId> = [new].into();
	963	+ let result = loop {
	964	+ let Some(id) = queue.pop_front() else {
	965	+ out.reverse();
	966	+ break CommitRange::Commits(out);
	967	+ };
	968	+ if hidden.contains(&id) {
	969	+ continue;
	970	+ }
	971	+ self.reads += 1;
	972	+ let Some(parents) = self.parents.get(&id).cloned() else {
	973	+ break CommitRange::Undecided;
	974	+ };
	975	+ out.push(RangeCommit {
	976	+ id,
	977	+ parents: parents.clone(),
	978	+ });
	979	+ if out.len() > max_commits {
	980	+ break CommitRange::TooLong;
	981	+ }
	982	+ queue.extend(parents.into_iter().filter(\|p\| seen.insert(*p)));
	983	+ };
	984	+ Box::pin(async move { Ok(result) })
	985	+ }
	986	+
	987	+ fn read(
	988	+ &mut self,
	989	+ id: ObjectId,
	990	+ ) -> HookFuture<'_, Result<Option<aspen_server::receive::hook::Object>, StoreError>>
	991	+ {
	992	+ self.reads += 1;
	993	+ let out = self.parents.get(&id).map(\|parents\| {
	994	+ let data = crate::git::write_commit(oid(0), parents, "T <t@example.com>", 0, "c");
	995	+ (ObjectKind::Commit, data)
	996	+ });
	997	+ Box::pin(async move { Ok(out) })
	998	+ }
	999	+ }
	1000	+
	1001	+ fn block_on<F: std::future::Future>(f: F) -> F::Output {
	1002	+ tokio::runtime::Builder::new_current_thread()
	1003	+ .build()
	1004	+ .unwrap()
	1005	+ .block_on(f)
	1006	+ }
	1007	+
	1008	+ /// One octopus generation of 5,000 parents, none of them indexed or
	1009	+ /// readable: the walk asks about and reads at most its budget in all,
	1010	+ /// never the whole generation, in batches the index can take.
	1011	+ #[test]
	1012	+ fn the_boundary_walk_stays_within_its_budget_on_a_wide_generation() {
	1013	+ let budget = 1000;
	1014	+ let index = Index::default();
	1015	+ let mut objects = Objects::default();
	1016	+ let new = oid(1);
	1017	+ objects.add(new, (10..5010).map(oid).collect());
	1018	+ let boundary = block_on(main_boundary(&index, &mut objects, new, budget)).unwrap();
	1019	+ assert!(boundary.is_empty());
	1020	+ let spent = index.asked.load(Ordering::Relaxed) + objects.reads;
	1021	+ assert!(spent <= budget, "spent {spent} of {budget}");
	1022	+ assert!(spent >= budget - 1, "gave up early: {spent}");
	1023	+ assert!(index.largest_batch.load(Ordering::Relaxed) <= CLASSIFY_BATCH);
	1024	+ }
	1025	+
	1026	+ /// A chain of 10,000 indexed commits off `main` with `main` below it:
	1027	+ /// the walk gives up within its budget, and finds `main` when the
	1028	+ /// chain is short enough. Each version is looked up once per walk.
	1029	+ #[test]
	1030	+ fn the_boundary_walk_gives_up_on_a_deep_chain_off_main() {
	1031	+ for (chain, budget, found) in [(10_000usize, 1000usize, false), (100, 1000, true)] {
	1032	+ let mut index = Index::default();
	1033	+ let mut objects = Objects::default();
	1034	+ let main = oid(0);
	1035	+ index.rows.insert(main, Indexed::Main);
	1036	+ for n in 1..=chain {
	1037	+ index.rows.insert(oid(n), Indexed::Other(vec![oid(n - 1)]));
	1038	+ }
	1039	+ let new = oid(chain + 1);
	1040	+ objects.add(new, vec![oid(chain)]);
	1041	+ let boundary = block_on(main_boundary(&index, &mut objects, new, budget)).unwrap();
	1042	+ assert_eq!(boundary == vec![main], found, "chain {chain}");
	1043	+ let spent = index.asked.load(Ordering::Relaxed) + objects.reads;
	1044	+ assert!(spent <= budget, "chain {chain}: spent {spent} of {budget}");
	1045	+ }
	1046	+ }
	1047	+
	1048	+ /// A clone from before a history reset pushes three commits on the old
	1049	+ /// `main`: the hook reads and asks about the same few dozen commits
	1050	+ /// whether the old history is 200 or 5,000 commits long, never walking
	1051	+ /// it down to its root. An ordinary push on today's `main` asks the
	1052	+ /// index nothing.
	1053	+ #[test]
	1054	+ fn a_push_on_an_earlier_main_costs_the_same_whatever_its_length() {
	1055	+ let max_commits = 1000;
	1056	+ let mut costs = Vec::new();
	1057	+ for old in [200usize, 5000] {
	1058	+ let mut index = Index::default();
	1059	+ let mut objects = Objects::default();
	1060	+ // The old history: a chain from its root, all on `main`.
	1061	+ objects.add(oid(0), Vec::new());
	1062	+ index.rows.insert(oid(0), Indexed::Main);
	1063	+ for n in 1..old {
	1064	+ objects.add(oid(n), vec![oid(n - 1)]);
	1065	+ index.rows.insert(oid(n), Indexed::Main);
	1066	+ }
	1067	+ // Today's `main`: a root of its own.
	1068	+ let today = oid(1_000_000);
	1069	+ objects.add(today, Vec::new());
	1070	+ index.rows.insert(today, Indexed::Main);
	1071	+ // The push: three commits on the old tip.
	1072	+ let pushed = [oid(2_000_001), oid(2_000_002), oid(2_000_003)];
	1073	+ objects.add(pushed[0], vec![oid(old - 1)]);
	1074	+ objects.add(pushed[1], vec![pushed[0]]);
	1075	+ objects.add(pushed[2], vec![pushed[1]]);
	1076	+ let range = block_on(commit_range(
	1077	+ &index,
	1078	+ &mut objects,
	1079	+ pushed[2],
	1080	+ vec![today],
	1081	+ max_commits,
	1082	+ ))
	1083	+ .unwrap();
	1084	+ let CommitRange::Commits(commits) = range else {
	1085	+ panic!("old {old}: {range:?}");
	1086	+ };
	1087	+ let ids: Vec<ObjectId> = commits.iter().map(\|c\| c.id).collect();
	1088	+ assert_eq!(ids, pushed, "old {old}");
	1089	+ let cost = index.asked.load(Ordering::Relaxed) + objects.reads;
	1090	+ assert!(cost <= PROBE_COMMITS + 16, "old {old}: cost {cost}");
	1091	+ costs.push(cost);
	1092	+
	1093	+ // An ordinary push: two commits on today's main.
	1094	+ let mut objects2 = Objects::default();
	1095	+ objects2.add(today, Vec::new());
	1096	+ objects2.add(oid(3_000_001), vec![today]);
	1097	+ objects2.add(oid(3_000_002), vec![oid(3_000_001)]);
	1098	+ let before = index.asked.load(Ordering::Relaxed);
	1099	+ let range = block_on(commit_range(
	1100	+ &index,
	1101	+ &mut objects2,
	1102	+ oid(3_000_002),
	1103	+ vec![today],
	1104	+ max_commits,
	1105	+ ))
	1106	+ .unwrap();
	1107	+ assert!(matches!(range, CommitRange::Commits(ref c) if c.len() == 2));
	1108	+ assert_eq!(index.asked.load(Ordering::Relaxed), before, "no lookups");
	1109	+ assert_eq!(objects2.reads, 2);
	1110	+ }
	1111	+ assert_eq!(costs[0], costs[1], "cost grew with the old history");
	1112	+ }
	1113	+}

demo/place/server/src/rules.rs+12−0

@@ -13,6 +13,18 @@ pub const TRUNK_REF: &str = "refs/heads/main"; · 12 unmodified lines
13	13	/// visitor's branch only ever moves through that visitor's session.
14	14	pub const RESERVED_PREFIXES: [&str; 3] = ["visitors/", "agent/", "swarm/"];
15	15
	16	+/// Branches under this prefix (after `refs/heads/`) only move forward
	17	+/// for Git clients: they may be created and fast-forwarded, never deleted
	18	+/// or rewritten, because anyone may push to any of them.
	19	+pub const FORWARD_ONLY_PREFIX: &str = "paint/";
	20	+
	21	+/// Whether a Git client's update of `name` must be a creation or a
	22	+/// fast-forward ([`FORWARD_ONLY_PREFIX`]).
	23	+pub fn forward_only(name: &str) -> bool {
	24	+ name.strip_prefix("refs/heads/")
	25	+ .is_some_and(\|short\| short.starts_with(FORWARD_ONLY_PREFIX))
	26	+}
	27	+
16	28	/// The only file mode a tile may have.
17	29	pub const TILE_MODE: u32 = 0o100644;
18	30

demo/place/server/tests/place.rs+146−5

@@ -253,18 +253,19 @@ async fn every_tile_rule_is_enforced_through_git_push_and_a_refused_push_stores_ · 252 unmodified lines
253	253
254	254	#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
255	255	async fn a_branch_is_checked_over_commits_new_to_it_not_from_the_client_old_value() {
256		− // TILES.md's worked example with a cap of 4 tiles per push.
	256	+ // TILES.md's worked example with a cap of 4 tiles per push, on a
	257	+ // branch outside `paint/` so its step 3 force-push is allowed.
257	258	let app = App::start(\|c\| c.rules.tiles_per_push = 4).await;
258	259	let w = app.clone_work("ada").await;
259	260	let t0 = w.git(&["rev-parse", "HEAD"]).trim().to_owned();
260	261	w.paint(&[(0, 0, RED), (1, 0, RED), (2, 0, RED)], "A1");
261		− assert!(w.push(&[&app.url(), "HEAD:refs/heads/paint/ada"]).await.0);
	262	+ assert!(w.push(&[&app.url(), "HEAD:refs/heads/ada"]).await.0);
262	263	// The merger moves trunk meanwhile.
263	264	app.place.merge_once().await.unwrap().unwrap();
264	265	// A2 on A1: the branch is not advertised, so git sends a zero old
265	266	// value; the server excludes its stored tip A1, so 2 tiles count.
266	267	w.paint(&[(3, 0, RED), (4, 0, RED)], "A2");
267		− let (ok, out) = w.push(&[&app.url(), "HEAD:refs/heads/paint/ada"]).await;
	268	+ let (ok, out) = w.push(&[&app.url(), "HEAD:refs/heads/ada"]).await;
268	269	assert!(ok, "counted A1 again: {out}");
269	270	// A3 alone paints 5 tiles: refused.
270	271	w.paint(
@@ -277,7 +278,7 @@ async fn a_branch_is_checked_over_commits_new_to_it_not_from_the_client_old_valu · 6 unmodified lines
277	278	],
278	279	"A3",
279	280	);
280		− let (ok, out) = w.push(&[&app.url(), "HEAD:refs/heads/paint/ada"]).await;
	281	+ let (ok, out) = w.push(&[&app.url(), "HEAD:refs/heads/ada"]).await;
281	282	assert!(
282	283	!ok && out.contains("this push changes 5 tiles; the limit is 4"),
283	284	"{out}"
@@ -294,7 +295,7 @@ async fn a_branch_is_checked_over_commits_new_to_it_not_from_the_client_old_valu · 10 unmodified lines
294	295	"B1",
295	296	);
296	297	let (ok, out) = w
297		− .push(&["-o", "force", &app.url(), "+HEAD:refs/heads/paint/ada"])
	298	+ .push(&["-o", "force", &app.url(), "+HEAD:refs/heads/ada"])
298	299	.await;
299	300	assert!(ok, "{out}");
300	301	let canvas = app.get("/api/v1/canvas").await;
@@ -303,6 +304,146 @@ async fn a_branch_is_checked_over_commits_new_to_it_not_from_the_client_old_valu · 2 unmodified lines
303	304	assert_eq!(overlay[0]["tiles"].as_array().unwrap().len(), 6);
304	305	}
305	306
	307	+/// Invariant: a Git client can create and fast-forward a `paint/`
	308	+/// branch, but cannot delete it or rewrite it, however it asks (`-o
	309	+/// force`, `--force` through the branch URL, a rewind to an ancestor); the
	310	+/// refusal tells it to push to a new branch, and the branch is untouched.
	311	+/// The server's own handling of `paint/` branches (merging, expiry's
	312	+/// delete) is not affected, and other branches keep Aspen's policy.
	313	+#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
	314	+async fn paint_branches_only_move_forward_for_git_clients() {
	315	+ let app = App::start(\|c\| c.branch_ttl = Duration::from_millis(300)).await;
	316	+ let bob = "refs/heads/paint/bob";
	317	+ let w = app.clone_work("bob").await;
	318	+ let t0 = w.git(&["rev-parse", "HEAD"]).trim().to_owned();
	319	+ let a1 = w.paint(&[(0, 0, RED)], "A1");
	320	+ let (ok, out) = w.push(&[&app.url(), &format!("HEAD:{bob}")]).await;
	321	+ assert!(ok, "creation refused: {out}");
	322	+ let a2 = w.paint(&[(1, 0, RED)], "A2");
	323	+ let (ok, out) = w.push(&[&app.url(), &format!("HEAD:{bob}")]).await;
	324	+ assert!(ok, "fast-forward refused: {out}");
	325	+ assert_eq!(app.tip(bob).await.as_deref(), Some(&a2[..]));
	326	+
	327	+ // Someone else's rewrite of the branch, on T0.
	328	+ let m = app.clone_work("mallory").await;
	329	+ m.git(&["reset", "-q", "--hard", &t0]);
	330	+ let b1 = m.paint(&[(5, 5, BLUE)], "B1");
	331	+ let scoped = format!("{}/-/{bob}", app.url());
	332	+ let forward = "paint/ branches only move forward";
	333	+ let attempts: [(&str, Vec<String>); 3] = [
	334	+ (
	335	+ "-o force",
	336	+ vec![
	337	+ "-o".into(),
	338	+ "force".into(),
	339	+ app.url(),
	340	+ format!("+HEAD:{bob}"),
	341	+ ],
	342	+ ),
	343	+ (
	344	+ "--force through the branch URL",
	345	+ vec!["--force".into(), scoped.clone(), format!("HEAD:{bob}")],
	346	+ ),
	347	+ (
	348	+ "a rewind to an ancestor",
	349	+ vec![
	350	+ "-o".into(),
	351	+ "force".into(),
	352	+ app.url(),
	353	+ format!("+{a1}:{bob}"),
	354	+ ],
	355	+ ),
	356	+ ];
	357	+ for (what, args) in &attempts {
	358	+ let args: Vec<&str> = args.iter().map(String::as_str).collect();
	359	+ let (ok, out) = m.push(&args).await;
	360	+ assert!(!ok, "{what} accepted: {out}");
	361	+ assert!(
	362	+ out.contains(forward) && out.contains("push your history to a new branch"),
	363	+ "{what}: {out}"
	364	+ );
	365	+ assert_eq!(app.tip(bob).await.as_deref(), Some(&a2[..]), "{what}");
	366	+ }
	367	+ // Without `-o force`, Aspen's own ref-update rule (spike S2) refuses
	368	+ // a delete or a rewrite of a branch this client was not shown, before
	369	+ // the place hook is asked; the branch is untouched either way.
	370	+ let plain: [(&str, Vec<String>); 2] = [
	371	+ ("a plain delete", vec![app.url(), format!(":{bob}")]),
	372	+ (
	373	+ "a plain --force",
	374	+ vec!["--force".into(), app.url(), format!("HEAD:{bob}")],
	375	+ ),
	376	+ ];
	377	+ for (what, args) in &plain {
	378	+ let args: Vec<&str> = args.iter().map(String::as_str).collect();
	379	+ let (ok, out) = m.push(&args).await;
	380	+ assert!(!ok, "{what} accepted: {out}");
	381	+ assert_eq!(app.tip(bob).await.as_deref(), Some(&a2[..]), "{what}");
	382	+ }
	383	+ let deletes: [(&str, Vec<String>); 2] = [
	384	+ (
	385	+ "a delete with -o force",
	386	+ vec!["-o".into(), "force".into(), app.url(), format!(":{bob}")],
	387	+ ),
	388	+ (
	389	+ "--delete through the branch URL",
	390	+ vec!["--delete".into(), scoped.clone(), bob.into()],
	391	+ ),
	392	+ ];
	393	+ for (what, args) in &deletes {
	394	+ let args: Vec<&str> = args.iter().map(String::as_str).collect();
	395	+ let (ok, out) = m.push(&args).await;
	396	+ assert!(!ok, "{what} accepted: {out}");
	397	+ assert!(
	398	+ out.contains("paint/ branches cannot be deleted") && out.contains("new branch"),
	399	+ "{what}: {out}"
	400	+ );
	401	+ assert_eq!(app.tip(bob).await.as_deref(), Some(&a2[..]), "{what}");
	402	+ }
	403	+
	404	+ // The rewrite goes to a new branch instead.
	405	+ let (ok, out) = m.push(&[&app.url(), "HEAD:refs/heads/paint/mallory"]).await;
	406	+ assert!(ok, "{out}");
	407	+ assert_eq!(
	408	+ app.tip("refs/heads/paint/mallory").await.as_deref(),
	409	+ Some(&b1[..])
	410	+ );
	411	+ // Outside `paint/`, a delete is still Aspen's ordinary policy.
	412	+ let (ok, out) = m.push(&[&app.url(), "HEAD:refs/heads/scratch"]).await;
	413	+ assert!(ok, "{out}");
	414	+ let (ok, out) = m
	415	+ .push(&["-o", "force", &app.url(), ":refs/heads/scratch"])
	416	+ .await;
	417	+ assert!(ok, "{out}");
	418	+ assert_eq!(app.tip("refs/heads/scratch").await, None);
	419	+
	420	+ // The merger folds both paint/ branches into main, and expiry deletes
	421	+ // them through receive-pack as before.
	422	+ app.place.merge_once().await.unwrap().expect("a merge");
	423	+ let check = app.clone_work("check").await;
	424	+ for (path, color) in [
	425	+ ("tiles/0_0.txt", RED),
	426	+ ("tiles/1_0.txt", RED),
	427	+ ("tiles/5_5.txt", BLUE),
	428	+ ] {
	429	+ assert_eq!(
	430	+ std::fs::read(check.path.join(path)).unwrap(),
	431	+ format!("{color}\n").as_bytes(),
	432	+ "{path}"
	433	+ );
	434	+ }
	435	+ tokio::time::sleep(Duration::from_millis(400)).await;
	436	+ // Both paint/ branches, and `scratch`, which the delete above left
	437	+ // to expiry to forget.
	438	+ assert_eq!(app.place.expire_once().await.unwrap(), 3);
	439	+ assert_eq!(app.tip(bob).await, None);
	440	+ assert_eq!(app.tip("refs/heads/paint/mallory").await, None);
	441	+ assert_eq!(app.get("/api/v1/branches").await["branches"], json!([]));
	442	+ // A name expiry freed can be created again.
	443	+ let (ok, out) = w.push(&[&app.url(), &format!("HEAD:{bob}")]).await;
	444	+ assert!(ok, "{out}");
	445	+}
	446	+
306	447	#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
307	448	async fn git_pushes_are_limited_per_address_and_branch() {
308	449	let app = App::start(\|c\| c.pushes_per_minute = 2).await;

deploy/terraform/aws/public-demo/README.md+17−10

@@ -276,9 +276,11 @@ Every line must say `ok`. Then, by hand: · 275 unmodified lines
276	276	### 8. The public demo
277	277
278	278	The same steps with `demo.tfvars` and `demo.backend.hcl` (`terraform init
279		−-reconfigure -backend-config=demo.backend.hcl`). `demo.tfvars.example` starts
280		−`place` with its kill switch engaged; release it through the API once the
281		−smoke test passes and someone is watching the budget. Making the demo
	279	+-reconfigure -backend-config=demo.backend.hcl`). `place` starts with its
	280	+kill switch released, on its first start and after every restart. Until
	281	+someone watches the budget, apply with `place_running = false`, or engage
	282	+the switch through the API (see [Operate it](#operate-it)) right after the
	283	+task starts. Making the demo
282	284	public (linking it anywhere) waits for the launch gate in docs/PLAN.md
283	285	section 7.
284	286
@@ -293,10 +295,10 @@ two do not idle at $230 together. · 8 unmodified lines
293	295	new one starts, so the page is down for about a minute; that is the price
294	296	of never running two merger bots at once.
295	297	The live demo's `demo.tfvars` and `demo.backend.hcl` are kept by the
296		− maintainers, outside the repository. They keep `place_kill_switch = true`, so
297		− every deploy starts `place` with the kill switch engaged: release it with
298		− the operator token (`{"engaged": false}` to `/api/v1/admin/kill-switch`)
299		− once the new task is up.
	298	+ maintainers, outside the repository. A restart starts `place` with its kill
	299	+ switch released, so a deploy needs no operator step afterwards; the
	300	+ environment has no setting that starts it engaged (one did, and every
	301	+ deploy left the public demo stopped until someone released it).
300	302	- Mirror the canvas to GitHub. Set `place_canvas_mirror = "owner/name"`
301	303	(an existing, empty repository) and apply. Put a credential with write
302	304	access to that repository only into the secret the
@@ -306,15 +308,20 @@ two do not idle at $230 together. · 3 unmodified lines
306	308	a fine-grained token with Contents read/write on that repository alone.
307	309	`aws secretsmanager put-secret-value --secret-id <arn> --secret-string file://cred`,
308	310	delete the local copy, and force a new deployment of the place service.
309		− The `canvas-mirror` sidecar fetches `main` from aspen-place at start and
310		− every `place_canvas_mirror_interval_seconds` (900) and pushes it when it
	311	+ The `canvas-mirror` sidecar fetches `main` from aspen-place as soon as
	312	+ place answers after a start (it asks every 5 seconds until then, for at
	313	+ most one interval) and then every
	314	+ `place_canvas_mirror_interval_seconds` (900), and pushes it when it
311	315	moved, forced, so the mirror follows `main` when place starts its
312	316	history again (`PLACE_HISTORY_MAX_BYTES`, or
313	317	`/api/v1/admin/reset-history`); its
314	318	log stream (`canvas-mirror/...` in the place log group) says when a push
315	319	fails.
316	320	- Stop the demo without tearing it down: engage the kill switch (API),
317		− or set `place_running = false` and apply, which stops the task.
	321	+ or set `place_running = false` and apply, which stops the task. The kill
	322	+ switch lives in the task's memory, so a restart or deploy releases it;
	323	+ engage it again afterwards, or use `place_running = false` for a stop
	324	+ that lasts.
318	325	- When a budget alert fires: look at Cost Explorer by service first.
319	326	- CloudFront: set `enable_cdn = false` in the environment's `.tfvars` and
320	327	apply. That removes the distribution, so CloudFront stops serving and

deploy/terraform/aws/public-demo/environment/canvas-mirror.sh+53−8

@@ -1,11 +1,14 @@
1	1	# shellcheck shell=sh
2		−# The canvas mirror sidecar (place.tf): at start and then every
3		−# MIRROR_INTERVAL seconds (15 minutes by default), fetch the canvas's main
	2	+# The canvas mirror sidecar (place.tf): once aspen-place answers after a
	3	+# start, and then every MIRROR_INTERVAL seconds (15 minutes by default),
	4	+# fetch the canvas's main
4	5	# from aspen-place in the same task and, when it moved, push it to the
5	6	# GitHub repository MIRROR_REPO: one push per run. The push is forced,
6	7	# because place starts main's history again from one root commit whenever
7	8	# it outgrows PLACE_HISTORY_MAX_BYTES; the GitHub repository is a read-only
8		−# copy whose only writer is this sidecar.
	9	+# copy whose only writer is this sidecar. It is forced with a lease: only
	10	+# over the main this sidecar last pushed (read from GitHub at start), so a
	11	+# write by anyone else is reported, never overwritten.
9	12	#
10	13	# MIRROR_CREDENTIAL is either an SSH deploy key (OpenSSH private key; SSH
11	14	# goes over port 443, ssh.github.com, the only egress the place task has)
@@ -32,18 +35,60 @@ case "$MIRROR_CREDENTIAL" in · 20 unmodified lines
32	35	;;
33	36	esac
34	37	unset MIRROR_CREDENTIAL
35		−git init --quiet --bare /tmp/canvas.git
36		−cd /tmp/canvas.git
	38	+git init --quiet --bare "${TMPDIR:-/tmp}/canvas.git"
	39	+cd "${TMPDIR:-/tmp}/canvas.git"
	40	+# This sidecar starts with place, which listens only once it has loaded the
	41	+# canvas. Ask place for main every MIRROR_START_RETRY seconds (5) until it
	42	+# names one, for at most one MIRROR_INTERVAL, so the first push after a
	43	+# restart comes seconds after place is up rather than a whole interval
	44	+# later. Each ask is cut off after 10 seconds, so a place that accepts the
	45	+# connection and then stalls cannot hold the wait past its interval.
	46	+retry=${MIRROR_START_RETRY:-5}
	47	+case "$retry" in
	48	+ '' \| [!0-9]) retry=0 ;;
	49	+esac
	50	+if [ "$retry" -le 0 ]; then
	51	+ echo "canvas-mirror: MIRROR_START_RETRY=${MIRROR_START_RETRY:-} is not a positive number of seconds; using 5"
	52	+ retry=5
	53	+fi
	54	+started=$(date +%s)
	55	+waiting=no
	56	+until [ -n "$(timeout 10 git ls-remote "$MIRROR_SOURCE" refs/heads/main 2>/dev/null)" ]; do
	57	+ waited=$(($(date +%s) - started))
	58	+ if [ "$waited" -ge "$MIRROR_INTERVAL" ]; then
	59	+ echo "canvas-mirror: place still does not answer at $MIRROR_SOURCE after ${waited}s"
	60	+ break
	61	+ fi
	62	+ if [ "$waiting" = no ]; then
	63	+ echo "canvas-mirror: waiting for place at $MIRROR_SOURCE"
	64	+ waiting=yes
	65	+ fi
	66	+ sleep "$retry"
	67	+done
	68	+# GitHub's main as this sidecar last left it ("" if it has none), once
	69	+# read.
37	70	pushed=""
	71	+known=no
38	72	while :; do
39		− if git fetch --quiet "$MIRROR_SOURCE" +refs/heads/main:refs/heads/main; then
	73	+ if [ "$known" = no ] && remote_main=$(git ls-remote "$remote" refs/heads/main); then
	74	+ pushed=$(printf '%s' "$remote_main" \| cut -f1)
	75	+ known=yes
	76	+ fi
	77	+ if [ "$known" = no ]; then
	78	+ echo "canvas-mirror: could not read main from $MIRROR_REPO"
	79	+ elif git fetch --quiet "$MIRROR_SOURCE" +refs/heads/main:refs/heads/main; then
40	80	tip=$(git rev-parse refs/heads/main)
41	81	if [ "$tip" != "$pushed" ]; then
42		− if git push --quiet "$remote" +refs/heads/main:refs/heads/main; then
	82	+ if git push --quiet --force-with-lease="refs/heads/main:$pushed" "$remote" refs/heads/main:refs/heads/main; then
43	83	echo "canvas-mirror: pushed main $tip to $MIRROR_REPO"
44	84	pushed=$tip
45	85	else
46		− echo "canvas-mirror: push of $tip to $MIRROR_REPO failed (GitHub refused)"
	86	+ echo "canvas-mirror: push of $tip to $MIRROR_REPO failed (GitHub refused, or its main is not ${pushed:-absent}, as this mirror left it)"
	87	+ # The push may have landed all the same; a main someone else
	88	+ # wrote stays theirs until the sidecar restarts.
	89	+ if [ "$(git ls-remote "$remote" refs/heads/main \| cut -f1)" = "$tip" ]; then
	90	+ pushed=$tip
	91	+ fi
47	92	fi
48	93	fi
49	94	else

deploy/terraform/aws/public-demo/environment/demo.tfvars.example+0−4

@@ -18,8 +18,4 @@ ecr_repository_arns = [ · 17 unmodified lines
18	18	# aspen_image = "123456789012.dkr.ecr.us-east-1.amazonaws.com/aspen:v0.1.0"
19	19	# place_image = "123456789012.dkr.ecr.us-east-1.amazonaws.com/aspen-place:v0.1.0"
20	20
21		−# First start with the swarm and ambient agents stopped; an operator
22		−# releases the kill switch once the dashboards are watched.
23		−place_kill_switch = true
24		−
25	21	enable_ssh = false

deploy/terraform/aws/public-demo/environment/place.tf+0−1

@@ -36,7 +36,6 @@ locals { · 35 unmodified lines
36	36	PLACE_DASHBOARD_URL = var.place_dashboard_url
37	37	PLACE_TRUSTED_PROXY_HOPS = "1"
38	38	PLACE_DB_MAX_CONNECTIONS = tostring(local.place_db_max_connections)
39		− PLACE_KILL_SWITCH = var.place_kill_switch ? "1" : ""
40	39	PLACE_MIRROR_URL = var.place_canvas_mirror == null ? "" : "https://github.com/${var.place_canvas_mirror}"
41	40	}, var.place_environment)
42	41	}

deploy/terraform/aws/public-demo/environment/tests/canvas-mirror.test.sh+150−0

@@ -0,0 +1,150 @@
	1	+#!/bin/sh
	2	+# Runs canvas-mirror.sh (the place task's mirror sidecar) against local
	3	+# repositories:
	4	+#
	5	+# sh deploy/terraform/aws/public-demo/environment/tests/canvas-mirror.test.sh
	6	+#
	7	+# Place is a bare repository that appears only some seconds after the
	8	+# sidecar starts, the way aspen-place listens only once it has loaded the
	9	+# canvas; GitHub is a bare repository that a `git` wrapper puts in place of
	10	+# https://github.com/<repo>.git (and `apk` does nothing). Checks that:
	11	+#
	12	+# 1. with a 15-minute MIRROR_INTERVAL, main is on "GitHub" within seconds of
	13	+# place answering, not an interval later (the sidecar used to fetch once
	14	+# at start, fail, and wait the whole interval);
	15	+# 2. when place never answers, the sidecar stops waiting after one interval
	16	+# and carries on with its usual runs;
	17	+# 3. a MIRROR_START_RETRY that is not a positive number of seconds is
	18	+# reported and replaced by 5, and the wait still ends once place answers;
	19	+# 4. a place that accepts the request and never answers (here a `git` that
	20	+# sleeps) cannot hold the wait: each ask is cut off after 10 seconds.
	21	+#
	22	+# Needs git, pkill and timeout.
	23	+set -eu
	24	+
	25	+here=$(cd "$(dirname "$0")" && pwd)
	26	+script="$here/../canvas-mirror.sh"
	27	+real_git=$(command -v git)
	28	+
	29	+dir=$(mktemp -d)
	30	+pids=""
	31	+cleanup() {
	32	+ # The sidecar's children first (its sleep), then the sidecar.
	33	+ for pid in $pids; do
	34	+ pkill -P "$pid" 2>/dev/null \|\| true
	35	+ kill "$pid" 2>/dev/null \|\| true
	36	+ done
	37	+ rm -rf "$dir"
	38	+}
	39	+trap cleanup EXIT
	40	+
	41	+mkdir -p "$dir/bin"
	42	+printf '#!/bin/sh\nexit 0\n' > "$dir/bin/apk"
	43	+cat > "$dir/bin/git" <<EOF
	44	+#!/bin/sh
	45	+# Every argument naming the GitHub repository names the local stand-in.
	46	+# A git naming the stalled place hangs, as a place that took the request
	47	+# and never answered would.
	48	+for a in "\$@"; do
	49	+ shift
	50	+ if [ "\$a" = "https://github.com/example/place-canvas.git" ]; then a="\$GITHUB_STANDIN"; fi
	51	+ if [ -n "\${STALLED_PLACE:-}" ] && [ "\$a" = "\$STALLED_PLACE" ]; then exec sleep 600; fi
	52	+ set -- "\$@" "\$a"
	53	+done
	54	+exec "$real_git" "\$@"
	55	+EOF
	56	+chmod +x "$dir/bin/apk" "$dir/bin/git"
	57	+
	58	+fail() {
	59	+ echo "FAIL: $*" >&2
	60	+ for log in "$dir"/*.log; do
	61	+ [ -f "$log" ] \|\| continue
	62	+ echo "--- $log" >&2
	63	+ cat "$log" >&2
	64	+ done
	65	+ exit 1
	66	+}
	67	+
	68	+# start_sidecar <name> <interval> <place> [<start retry>] [<stalled place>]:
	69	+# starts the sidecar (start retry 1 s by default), logging to
	70	+# $dir/<name>.log, with GitHub at $dir/<name>-github.git (empty).
	71	+start_sidecar() {
	72	+ "$real_git" init --quiet --bare "$dir/$1-github.git"
	73	+ mkdir -p "$dir/$1-home" "$dir/$1-tmp"
	74	+ env -u GIT_CONFIG_GLOBAL -u GIT_CONFIG_SYSTEM \
	75	+ PATH="$dir/bin:$PATH" HOME="$dir/$1-home" TMPDIR="$dir/$1-tmp" GIT_CONFIG_NOSYSTEM=1 \
	76	+ GITHUB_STANDIN="$dir/$1-github.git" \
	77	+ MIRROR_REPO=example/place-canvas MIRROR_CREDENTIAL=test-token \
	78	+ MIRROR_SOURCE="$3" MIRROR_INTERVAL="$2" MIRROR_START_RETRY="${4-1}" STALLED_PLACE="${5:-}" \
	79	+ sh "$script" > "$dir/$1.log" 2>&1 &
	80	+ pids="$pids $!"
	81	+}
	82	+
	83	+# wait_for <seconds> <command...>: true once the command succeeds.
	84	+wait_for() {
	85	+ n=$1
	86	+ shift
	87	+ while [ "$n" -gt 0 ]; do
	88	+ if "$@"; then return 0; fi
	89	+ sleep 1
	90	+ n=$((n - 1))
	91	+ done
	92	+ "$@"
	93	+}
	94	+
	95	+# A place repository with one commit on main, appearing at $1 all at once.
	96	+make_place() {
	97	+ "$real_git" init --quiet --bare "$1.new"
	98	+ work="$dir/work-$(basename "$1")"
	99	+ "$real_git" init --quiet "$work"
	100	+ (
	101	+ cd "$work"
	102	+ printf 'canvas\n' > tile
	103	+ "$real_git" add tile
	104	+ "$real_git" -c user.name=place -c user.email=place@example.com commit --quiet -m canvas
	105	+ "$real_git" push --quiet "$1.new" HEAD:refs/heads/main
	106	+ )
	107	+ mv "$1.new" "$1"
	108	+}
	109	+
	110	+github_main() { "$real_git" --git-dir="$1" rev-parse --verify --quiet refs/heads/main \|\| true; }
	111	+
	112	+# 1. Place comes up 3 seconds after the sidecar; the interval is 15 minutes.
	113	+start_sidecar late 900 "$dir/late-place.git"
	114	+sleep 3
	115	+make_place "$dir/late-place.git"
	116	+tip=$("$real_git" --git-dir="$dir/late-place.git" rev-parse refs/heads/main)
	117	+started=$(date +%s)
	118	+wait_for 20 grep -qs "canvas-mirror: pushed main $tip" "$dir/late.log" \|\|
	119	+ fail "main was not pushed within 20 seconds of place answering (interval 900 s)"
	120	+took=$(($(date +%s) - started))
	121	+[ "$(github_main "$dir/late-github.git")" = "$tip" ] \|\| fail "GitHub's main is not place's main"
	122	+grep -qs "canvas-mirror: waiting for place" "$dir/late.log" \|\| fail "the wait was not logged"
	123	+if grep -qs "fetch from .* failed" "$dir/late.log"; then fail "a fetch failed while place was starting"; fi
	124	+echo "ok: main mirrored ${took}s after place answered (interval 900 s)"
	125	+
	126	+# 2. Place never answers; the interval is 3 seconds.
	127	+start_sidecar never 3 "$dir/never-place.git"
	128	+wait_for 20 grep -qs "fetch from $dir/never-place.git failed" "$dir/never.log" \|\|
	129	+ fail "the sidecar did not carry on after waiting one interval"
	130	+grep -Eqs "canvas-mirror: place still does not answer at $dir/never-place.git after [0-9]+s" "$dir/never.log" \|\|
	131	+ fail "giving up the wait was not logged"
	132	+echo "ok: the wait ends after one interval and the usual runs go on"
	133	+
	134	+# 3. A start retry of 0 (which used to spin without counting time) is
	135	+# replaced by 5; place comes up 3 seconds after the sidecar.
	136	+start_sidecar zero 900 "$dir/zero-place.git" 0
	137	+sleep 3
	138	+make_place "$dir/zero-place.git"
	139	+tip=$("$real_git" --git-dir="$dir/zero-place.git" rev-parse refs/heads/main)
	140	+wait_for 20 grep -qs "canvas-mirror: pushed main $tip" "$dir/zero.log" \|\|
	141	+ fail "with MIRROR_START_RETRY=0, main was not pushed within 20 seconds of place answering"
	142	+grep -qs "canvas-mirror: MIRROR_START_RETRY=0 is not a positive number of seconds; using 5" "$dir/zero.log" \|\|
	143	+ fail "MIRROR_START_RETRY=0 was not reported"
	144	+echo "ok: MIRROR_START_RETRY=0 falls back to 5 s and main is mirrored"
	145	+
	146	+# 4. Place takes the request and never answers; the interval is 3 seconds.
	147	+start_sidecar stalled 3 "$dir/stalled-place.git" 1 "$dir/stalled-place.git"
	148	+wait_for 25 grep -Eqs "canvas-mirror: place still does not answer at $dir/stalled-place.git after [0-9]+s" "$dir/stalled.log" \|\|
	149	+ fail "a stalled place held the wait past its interval"
	150	+echo "ok: a stalled place cannot hold the wait"

deploy/terraform/aws/public-demo/environment/tests/environment.tftest.hcl+8−0

@@ -277,6 +277,14 @@ run "the_demo_can_serve_place_at_its_zone_apex" { · 276 unmodified lines
277	277	)
278	278	error_message = "The demo's place pool must have 64 connections."
279	279	}
	280	+ # The kill switch lives in place's memory: an env var that engages it at
	281	+ # start would leave the public demo stopped after every restart.
	282	+ assert {
	283	+ condition = alltrue([
	284	+ for e in jsondecode(aws_ecs_task_definition.place[0].container_definitions)[1].environment : e.name != "PLACE_KILL_SWITCH"
	285	+ ])
	286	+ error_message = "place must start with its kill switch released: no PLACE_KILL_SWITCH in its environment."
	287	+ }
280	288	assert {
281	289	condition = (
282	290	module.aspen.database_size.instance_class == "db.m7g.2xlarge" &&

deploy/terraform/aws/public-demo/environment/variables.tf+0−6

@@ -188,12 +188,6 @@ variable "place_live_cdn" { · 187 unmodified lines
188	188	default = true
189	189	}
190	190
191		−variable "place_kill_switch" {
192		− description = "Start place with its kill switch engaged (no swarm, ambient agents paused); an operator releases it through the API. Use it for the first start of the public demo."
193		− type = bool
194		− default = false
195		−}
196		−
197	191	variable "place_cpu" {
198	192	description = "Fargate CPU units for the place task (1024 = 1 vCPU). null takes the environment's size from main.tf (demo 16384, staging 512)."
199	193	type = number

docs/PLAN.md+13−10

@@ -600,7 +600,7 @@ full rule table and the decisive wire excerpts are in · 599 unmodified lines
600	600	step always runs one), so it collects everything due without holding up
601	601	jobs.
602	602	- The same worker runs `index_commits` jobs (commit rows for large pushes).
603		−- Keeping up (decisions log, 2026-11-07): a job reads only the covers
	603	+- Keeping up (decisions log, 2026-11-06): a job reads only the covers
604	604	inside the highest cover it builds (keyed, `covers_at`), never all of
605	605	its ref's covers, so its metadata work does not grow with the ref's
606	606	history; `ASPEN_ROLE` (`all`, `serve`, `compact`) runs the worker in its
@@ -1257,19 +1257,22 @@ Launch checklist for that: · 650 unmodified lines
1257	1257	\| 2026-11-06 \| Nightly red (dedupe key `aspen-nightly-red-2026-11`). (1) The differential oracle runs Aspen without per-caller concurrency caps: `aspen-testkit`'s `AspenServer` sets `ASPEN_LIMITS` to `aspen::ORACLE_LIMITS` (`concurrent` fetches, pushes and stitches `null`); request rates, quotas and the instance-wide limits keep their defaults. (2) A chaos epoch waits for its faults: while a composite injector (`mixed`, `smoke`) has a fault kind that never injected, the epoch's active phase goes on past `--epoch`, for at most `RunConfig::fault_coverage_limit` (5 minutes) from its start (`ActivePhase::wait_for_faults`); a kind still idle then fails the run as vacuous, as before. (3) The client-matrix Git image (`testkit/images/git`) downloads the release tarball to a file, from `mirrors.edge.kernel.org` then `cdn.kernel.org`, three rounds, each transfer time-bounded and retried; `run-matrix.sh` resolves `git-latest` the same way \| (1) Every oracle client is one anonymous caller, `127.0.0.1`, and `aspen serve` caps a caller at 4 concurrent pushes by default (issue #118). The nightly `aspen, sqlite, wide` row (5 clones) runs up to 5 concurrent pushes, so when all 5 overlapped one was refused with `429` before it reported a ref, and the oracle saw `Push { ok: false, refs: [] }` against stock Git's rejected ref (nightly runs 41961210154 and 42156103215). The refusal is admission working as designed, and stock `git http-backend` has no counterpart, so the oracle compares ref semantics with the cap off; admission keeps its own tests in `aspen-cli` (`serve_bounds_concurrent_fetches_and_pushes_per_caller`). Test: `crates/aspen-cli/tests/differential.rs` `the_oracle_server_admits_more_concurrent_pushes_than_the_default_cap` holds 7 pushes of the oracle's address open and checks an oracle push still agrees with stock Git; before the fix it failed with `too many concurrent pushes for this caller (at most 4)`, and `aspen-testkit diff --seed 0x5eed0001 --clones 8 --branches 8 --candidate aspen` diverged at step 2. (2) The vacuity check is right: a run must exercise every fault it claims. But nightly `mixed` gets two 30-second epochs, and a Postgres failover waits for a new synchronous standby, so two failovers used up the steps and `kill` and `compaction kill` never came up (nightly run 42156103215, 5 faults in all). Waiting for coverage follows `ActiveUntil`: what a run must exercise happens however slow the machine is, within a bound. (3) A bare `curl \| tar` from one kernel.org mirror failed four jobs of nightly run 41961210154 (two after 18 minutes), with no Aspen code run; the required smoke already handles kernel.org the same way (`install-latest-git.sh`, issue #189) \|
1258	1258	\| 2026-11-06 \| B1 revised: the prefetch gate reserves a floor, not the full per-push budget. The gate is a semaphore of MiB over `prefetch_memory_bytes`. A slot first reserves `ReceiveConfig::prefetch_floor_bytes` (64 MiB) and reads under that budget (`StoredPacks::prefetch_bases_within`). On `PrefetchError::OverBudget` it gives the floor back, waits for the full `LocateOptions::max_prefetch_bytes` reservation (capped at the total) and reads again; over the full budget the push is refused as before. The default 4 GiB then admits 64 small pushes, not 2 \| Reserving the full budget (2 GiB with the default 512 MiB object limit) for every push admitted 2 pushes at once whatever their size. In the throughput benchmark (bench/throughput) that held a c7g.2xlarge to about 15 pushes/s with its CPU idle, while the server's memory at 375 pushes/s stayed under 1 GB. Releasing before waiting keeps the gate deadlock-free when every holder needs to grow. Tests: `the_defaults_admit_many_small_pushes` (fails with a full reservation per push: 2 slots), `slots_grow_to_the_full_budget_one_at_a_time` (no deadlock; one full holder at a time), `a_full_budget_over_the_total_is_capped`, `a_push_whose_bases_exceed_the_floor_grows_its_slot`; the earlier gate tests still pass \|
1259	1259	\| 2026-11-06 \| Merge queue (`aspen_server::queue`, `ASPEN_MERGE_QUEUE`, off by default). Clients push a change to their own ref under `refs/queue/<trunk branch>/` through the plain URL (an ordinary branch-like write; the namespace is writable only while the queue is on, and only the queue writes the outcome namespaces). A worker in every instance takes pending entries oldest first, up to `ASPEN_MERGE_QUEUE_BATCH`; for each it finds the base (first commit on the entry's first-parent line that is an ancestor of trunk, by the push path's exact ancestry walk), diffs base→entry and base→trunk per file (a directory that appears or goes away is its files appearing or going away, so changes adding different files to one new directory both land), rejects the entry if a path equals or is nested in one trunk or an earlier batch entry changed, and otherwise writes one trunk commit (entry's tree changes applied to the batch's tree, author and message kept, committer the queue; several commits squash into one). The batch lands as one `--atomic` in-process receive-pack push: trunk T→new tip, each queue ref deleted against the commit the worker read, each outcome ref created (`refs/queue-done/...` at the landed commit, `refs/queue-rejected/...` at a report commit on trunk naming the original commit and the conflicting paths) \| Unbatched landings are bounded by one fetch-rebase-push cycle per trunk update, however many clients push (bench/throughput); folding every pending change into one compare-and-swap removes that bound without changing the push model: trunk still moves only by fast-forward through receive-pack, and the queue's writes are checked, completed and stored like any push. Exactly once: the atomic push lands a batch together with its queue-ref deletes or not at all, and each delete is a compare-and-swap on the commit the batch used, so a lost race (a direct trunk push, another instance's worker, a client re-pushing its entry) refuses the whole batch and it is rebuilt from what is stored; no single-runner assumption. Two workers that build the same batch in the same second write identical commits, and the second push is a no-op replay (issue #132). The rejection report is a commit on trunk rather than a tag of the rejected commit, because the in-process push depends on trunk only and may not name objects only the deleted queue ref's closure holds. Tests: `crates/aspen-server/tests/merge_queue.rs` (one batch of N entries with stock-git checks and `fsck --strict`; a conflict rejected with its path while others land; delete-vs-modify and below-a-deleted-directory conflicts; same-batch overlap, squash, deletion and new directories; a deterministic lost compare-and-swap via a push hook; two racing workers and a direct push; queue off refuses `refs/queue/`) \|
1260		−\| 2026-11-07 \| Group commit (`aspen_server::receive::GroupCommit`, `ASPEN_GROUP_COMMIT`, off by default; `ASPEN_GROUP_COMMIT_WINDOW_MS` 2, `ASPEN_GROUP_COMMIT_MAX_BATCH` 64). Concurrent pushes share their two metadata write transactions: pack registrations that arrive within the window of the first go to `MetaStore::register_packs` and ref commits to `MetaStore::commit_pushes`, each one transaction per batch (at most 4 batches of each kind at once; requests that arrive while all run wait for the next, so batches grow with load). Inside the transaction every push (or pack) runs behind its own savepoint, so a lost compare-and-swap, a refused pack or a swept pack undoes that push alone and it gets its own answer (a lost one recomputes its verdict and is submitted again, as before); pushes run in the order of their first ref name and packs in checksum order, so concurrent batches cannot deadlock. A savepoint statement or `COMMIT` that fails fails the whole batch, and every push in it is refused. A push submits its commit only after its pack, `.idx` and entry table are uploaded, and is answered only from a batch that committed, so durable-before-acknowledged holds unchanged. Pack objects are not batched: each push still writes its own three objects \| One S3 object per batch would make every version in it name a pack holding objects its tip does not reach, so its `closure_exact` would be false (fetches could no longer use it as a have; a trunk version would be `wider`) and compaction would copy every batch-mate into each ref's covers; an object shared by several `packs` rows would also need offsets in every reader and reference counting in garbage collection. Cursor's design (the comparison Aspen's throughput is held to, https://cursor.com/blog/git-at-any-scale) also stores each push as its own object and batches only the write that publishes pushes, which in Aspen is the metadata transaction; the uploads of concurrent pushes already run in parallel and never serialize. Measured with 32 concurrent pushes to distinct branches, stock git, SQLite (`crates/aspen-server/tests/group_commit.rs`): 64 write transactions off in every run, 6 to 13 with a 20 ms window and 25 to 29 with the 2 ms default (three runs on a shared 4-vCPU machine, where push latency was too noisy to compare); 96 blob writes in every mode. Off by default until a Postgres and S3 benchmark shows the window pays for its latency. Tests: `crates/aspen-meta/tests/store.rs` (per-push isolation in one transaction, including an atomic push undone as a whole; per-pack isolation; concurrent batches over shared refs in opposite orders without deadlock, which fails with "deadlock detected" on Postgres if the ordering is removed; a deferred-trigger `COMMIT` failure on Postgres that lands nothing) and `crates/aspen-server/tests/group_commit.rs` (the measurement; one winner per raced branch inside batches; a failed batch refuses every push, and a store wrapper checks every named pack is uploaded before any commit) \|
	1260	+\| 2026-11-06 \| Group commit (`aspen_server::receive::GroupCommit`, `ASPEN_GROUP_COMMIT`, off by default; `ASPEN_GROUP_COMMIT_WINDOW_MS` 2, `ASPEN_GROUP_COMMIT_MAX_BATCH` 64). Concurrent pushes share their two metadata write transactions: pack registrations that arrive within the window of the first go to `MetaStore::register_packs` and ref commits to `MetaStore::commit_pushes`, each one transaction per batch (at most 4 batches of each kind at once; requests that arrive while all run wait for the next, so batches grow with load). Inside the transaction every push (or pack) runs behind its own savepoint, so a lost compare-and-swap, a refused pack or a swept pack undoes that push alone and it gets its own answer (a lost one recomputes its verdict and is submitted again, as before); pushes run in the order of their first ref name and packs in checksum order, so concurrent batches cannot deadlock. A savepoint statement or `COMMIT` that fails fails the whole batch, and every push in it is refused. A push submits its commit only after its pack, `.idx` and entry table are uploaded, and is answered only from a batch that committed, so durable-before-acknowledged holds unchanged. Pack objects are not batched: each push still writes its own three objects \| One S3 object per batch would make every version in it name a pack holding objects its tip does not reach, so its `closure_exact` would be false (fetches could no longer use it as a have; a trunk version would be `wider`) and compaction would copy every batch-mate into each ref's covers; an object shared by several `packs` rows would also need offsets in every reader and reference counting in garbage collection. Cursor's design (the comparison Aspen's throughput is held to, https://cursor.com/blog/git-at-any-scale) also stores each push as its own object and batches only the write that publishes pushes, which in Aspen is the metadata transaction; the uploads of concurrent pushes already run in parallel and never serialize. Measured with 32 concurrent pushes to distinct branches, stock git, SQLite (`crates/aspen-server/tests/group_commit.rs`): 64 write transactions off in every run, 6 to 13 with a 20 ms window and 25 to 29 with the 2 ms default (three runs on a shared 4-vCPU machine, where push latency was too noisy to compare); 96 blob writes in every mode. Off by default until a Postgres and S3 benchmark shows the window pays for its latency. Tests: `crates/aspen-meta/tests/store.rs` (per-push isolation in one transaction, including an atomic push undone as a whole; per-pack isolation; concurrent batches over shared refs in opposite orders without deadlock, which fails with "deadlock detected" on Postgres if the ordering is removed; a deferred-trigger `COMMIT` failure on Postgres that lands nothing) and `crates/aspen-server/tests/group_commit.rs` (the measurement; one winner per raced branch inside batches; a failed batch refuses every push, and a store wrapper checks every named pack is uploaded before any commit) \|
1261	1261	\| 2026-11-06 \| Throughput item 2: no per-push work grows with the repository. (1) Changed paths. After completion, `aspen_pack::changed::changed_paths` diffs each pushed commit's tree against its first parent's and descends only into entries that differ. An entry equal to the parent's under the same name exists (every committed version is connected), so it is neither looked up nor checked; an entry that differs is in the pushed pack (entered, if a tree) or goes to the connectivity check; a parent outside the pack is known when it is the tip of a dependency (trunk's, the scoped ref's) and checked otherwise. A pushed object equal to an entry reached from a dependency's tip is held without a lookup, as completion bases are. Only the remaining objects are looked up (held objects, the "already stored" test of #203, unchanged), and objects the walk does not reach (tags, completion bases, stray objects) have their references checked as before. Merges are diffed against their first parent only; an unreadable old tree counts as empty; a push whose trees exceed 16 MiB falls back to the whole-pack lookup. Refines the S2 connectivity rows: the check still covers exactly the objects the pack references and lacks, minus those proven present by the diff. (2) Trunk's newest trees in memory: `ReceiveConfig::trunk_tree_cache_bytes` (`ASPEN_TRUNK_TREE_CACHE_BYTES`, default 64 MiB, 0 = none) keeps the trees committed trunk pushes bring and the old trees walks read, keyed by repository and ID; a hit only supplies bytes, never "held" (that comes from the walk reaching it from a dependency's tip). (3) Sampled `.idx`. An `.idx` over `LocateOptions::max_whole_idx_bytes` (`ASPEN_IDX_WHOLE_MAX_BYTES`, default 64 MiB, about 2.4 M objects) is read once, checked as strictly as a whole parse, and kept as its fanout plus one ID of every 256 (`aspen_pack::sampled`); a lookup reads the 256 IDs that can hold the object (5 KiB) and its offset, and an entry read in such a pack reads 16 KiB and grows until the zlib stream ends. Refines the A2 row on the `.idx` cache \| Measured by `crates/aspen-server/tests/large_repo_push_cost.rs` (required: 65,610 files; nightly, ignored: 301,806 files; both 8 directories deep, against a 4,608-file git.git-sized repository; 24 one-file pushes after a warm-up, `.idx` cache scaled to 1 MiB with indexes over 256 KiB sampled, so the large repository outgrows it as a Linux-sized one, about 7 M objects, outgrows the default 256 MiB). Before (`7cbb7c0` plus counters): 65,610 files 149 ms server CPU per push against 8.3 ms (17.9x), 6.4 MB of `.idx` read per push, 50 objects touched against 41; 301,806 files 635 ms (72.6x), 26.2 MB, 86 against 41 (2.10x). After: 10.0 ms and 10.4 ms against 9.2 ms (1.09x, 1.14x), 55 KB and 72 KB of `.idx`, 18 objects touched against 13 (1.38x); IDs located per push 11 against 11 (79 and 43 before, against 39). With warm caches and an `.idx` that fits, per-push cost was already flat (about 10 ms for 4,608 and 327,680 files); what grew was the whole-`.idx` read on every cache miss and the lookup of every entry of every tree a push touched. A server's first push still reads each search pack's `.idx` once (a sample once, for a large one). Unit tests: `aspen_pack::sampled` (lookups agree with a whole parse at every position, read size and offset width; every flipped byte refused), `tests::locate::a_sampled_idx_answers_exactly_as_a_whole_one` (OFS_DELTA and REF_DELTA packs, a 200 KB entry, absent IDs), `tests::changed` (a one-file change looks up nothing; moves, copies, a revert, a merge, a deletion, a non-tip base and a first push leave nothing unchecked that the base lacks) \|
1262		−\| 2026-11-07 \| Compaction keeps up at high push rates (throughput item 3). (1) A `compact` job reads only the covers inside the highest cover ending at its version, keyed by level (`covers_at`; `fanout^(k-l)` keys at level `l` below the top level `k`: 1 for the usual job, 273 for the level-3 job, 4,369 for the level-4 one), instead of every cover of its ref (`MetaStore::covers`), so per-job metadata work is amortized constant in the ref's length. (2) `ASPEN_ROLE=all\\|serve\\|compact` (default `all`): `serve` serves Git, the merge queue and mirror mode with no background worker; `compact` runs only the worker, answering `/healthz` and `/readyz` on `ASPEN_LISTEN` with no Git route, and refuses `ASPEN_SSH_LISTEN` and `ASPEN_MERGE_QUEUE=on`. The Helm chart's `compactor.enabled` adds a compactor Deployment (and the server pods run `serve`); the AWS module's `compactor_count` adds an ECS service with no load balancer. GCP, Azure and Hetzner keep `all`. (3) Fetch heat: migration 0008 adds `jobs.priority`; a served fetch counts, per ref, the versions whose own push pack its plan sends and the earliest of them (`upload_pack::heat`), each process sums them and writes them at most once a second from a spawned task (`MetaStore::add_compaction_heat`, a new required trait method), raising that ref's queued `compact` jobs at or after the earliest; claims order by `priority DESC`, then due time, then ID. (4) `aspen_compaction_backlog` (queued plus running `compact` jobs, a database gauge), `aspen_fetch_uncompacted_versions_total`, `aspen_compaction_heat_write_failures_total`, and the ticket alert `AspenCompactionBacklogGrowing` \| (1) Every job read all `~N/15` cover rows of a ref of `N` versions, so at any fixed push rate the worker eventually fell behind and the backlog grew without bound. `crates/aspen-compact/tests/backlog.rs` (64 trunk pushes per tick, each a real one-blob pack, one worker with a fixed budget of 400 metadata rows per tick, 80 ticks): on `0b41ba7` rows per job grew from 33 to 254, the worker fell behind at tick 19 and the backlog reached 111 jobs at tick 80, still growing; with (1) 33 rows per level-1 job at any length, the last quarter of jobs read 1.10x the first quarter's rows, and the backlog was 0 after every tick. (2) A worker on the Git servers competes with pushes for CPU, disk and the database pool, and Git servers were the only way to add compaction; jobs are leased and garbage collection is safe on any number of workers (pack generations), so a separate pool needs no coordination. Pinned by `serve_and_compact_roles_split_git_from_compaction` (aspen-cli: a `serve` process leaves the job queued and the backlog gauge at 1, a `compact` process sharing its database and bucket builds the cover, serves no Git, and the backlog reads 0) and the chart and Terraform checks. (3) With a backlog, FIFO builds covers nobody reads while fetches of hot refs send one pack per version. Heat is advisory (a lost write changes only order), batched to one write per process per second, and bounded (`MAX_PENDING_REFS`, `MAX_JOB_HEAT`). Cost: if hot refs alone exceed the workers, jobs of refs nobody fetches wait (the lag gauge shows it). Pinned by `heat_raises_the_queued_compact_jobs_fetches_wait_on` (store, both dialects) and `a_fetched_refs_compact_jobs_are_claimed_first` (stock git clones one branch; its job is claimed before an older one of a branch nobody fetched). The fetch path's own cover read (`upload_pack::graph` loads every cover of each ref it plans) is not changed here \|
	1262	+\| 2026-11-06 \| Compaction keeps up at high push rates (throughput item 3). (1) A `compact` job reads only the covers inside the highest cover ending at its version, keyed by level (`covers_at`; `fanout^(k-l)` keys at level `l` below the top level `k`: 1 for the usual job, 273 for the level-3 job, 4,369 for the level-4 one), instead of every cover of its ref (`MetaStore::covers`), so per-job metadata work is amortized constant in the ref's length. (2) `ASPEN_ROLE=all\\|serve\\|compact` (default `all`): `serve` serves Git, the merge queue and mirror mode with no background worker; `compact` runs only the worker, answering `/healthz` and `/readyz` on `ASPEN_LISTEN` with no Git route, and refuses `ASPEN_SSH_LISTEN` and `ASPEN_MERGE_QUEUE=on`. The Helm chart's `compactor.enabled` adds a compactor Deployment (and the server pods run `serve`); the AWS module's `compactor_count` adds an ECS service with no load balancer. GCP, Azure and Hetzner keep `all`. (3) Fetch heat: migration 0008 adds `jobs.priority`; a served fetch counts, per ref, the versions whose own push pack its plan sends and the earliest of them (`upload_pack::heat`), each process sums them and writes them at most once a second from a spawned task (`MetaStore::add_compaction_heat`, a new required trait method), raising that ref's queued `compact` jobs at or after the earliest; claims order by `priority DESC`, then due time, then ID. (4) `aspen_compaction_backlog` (queued plus running `compact` jobs, a database gauge), `aspen_fetch_uncompacted_versions_total`, `aspen_compaction_heat_write_failures_total`, and the ticket alert `AspenCompactionBacklogGrowing` \| (1) Every job read all `~N/15` cover rows of a ref of `N` versions, so at any fixed push rate the worker eventually fell behind and the backlog grew without bound. `crates/aspen-compact/tests/backlog.rs` (64 trunk pushes per tick, each a real one-blob pack, one worker with a fixed budget of 400 metadata rows per tick, 80 ticks): on `0b41ba7` rows per job grew from 33 to 254, the worker fell behind at tick 19 and the backlog reached 111 jobs at tick 80, still growing; with (1) 33 rows per level-1 job at any length, the last quarter of jobs read 1.10x the first quarter's rows, and the backlog was 0 after every tick. (2) A worker on the Git servers competes with pushes for CPU, disk and the database pool, and Git servers were the only way to add compaction; jobs are leased and garbage collection is safe on any number of workers (pack generations), so a separate pool needs no coordination. Pinned by `serve_and_compact_roles_split_git_from_compaction` (aspen-cli: a `serve` process leaves the job queued and the backlog gauge at 1, a `compact` process sharing its database and bucket builds the cover, serves no Git, and the backlog reads 0) and the chart and Terraform checks. (3) With a backlog, FIFO builds covers nobody reads while fetches of hot refs send one pack per version. Heat is advisory (a lost write changes only order), batched to one write per process per second, and bounded (`MAX_PENDING_REFS`, `MAX_JOB_HEAT`). Cost: if hot refs alone exceed the workers, jobs of refs nobody fetches wait (the lag gauge shows it). Pinned by `heat_raises_the_queued_compact_jobs_fetches_wait_on` (store, both dialects) and `a_fetched_refs_compact_jobs_are_claimed_first` (stock git clones one branch; its job is claimed before an older one of a branch nobody fetched). The fetch path's own cover read (`upload_pack::graph` loads every cover of each ref it plans) is not changed here \|
1263	1263	\| 2026-11-06 \| B1 refined (main red, dedupe key `aspen-main-red-branch-push-cost-2026-11-06`): the ancestry walk breaks committer-time ties by paint, then first in, first out. Among commits with equal committer times, a commit carrying both flags goes first, then the rest in the order they were queued, `b` (the would-be descendant) before `a`; a queued commit that gains both flags is queued again at that rank and its old entry skipped. Replaces the object-ID tie-break of the B1 ancestry row \| Agents commit and push several times a second, so a push's old and new tips usually share a committer time. With ties broken by object ID, the old tip's side of a fast-forward check walked down every same-second ancestor whose ID sorted above the new tip's: `branch_push_cost.rs` saw pushes walk up to 70 commits on CI (run 42187981485) and up to 34 locally, failing its average check about 1 run in 30. First in, first out moves both sides one generation at a time (a fast-forward by one commit expands one); both-flags-first lets the paint that ends the walk catch `b`'s side when `a` is the newer commit, which plain first in, first out leaves one generation behind down to the root. Order never affects the answer. Tests: `receive::ancestry` `equal_commit_times_cost_the_distance_not_the_history` (300 same-second commits with hash-like IDs: one expansion per one-commit fast-forward, at most 2d for distance d, at most 4 for the reverse question; fails at once with ID ties) and `answers_match_reachability_when_times_tie` (40 random DAGs with three distinct times, every pair against brute force); `branch_push_cost.rs` now also bounds every push at 4 commits (old walker: 10 of 10 runs fail; fixed: 30 of 30 pass, two at a time). Git avoids this with commit-graph generation numbers; `commit_index` has none, and with long same-second histories (scripted imports with a fixed date) a walk can still visit that second's commits \|
1264	1264	\| 2026-11-06 \| `push.negotiate` (throughput item 1: plain-URL pushes as cheap as the branch-scoped URL). (1) The v2 advertisement says `fetch=shallow wait-for-done` and the parser accepts `wait-for-done` (`FetchArgs::wait_for_done`), which removes it from the refused list of the A1 strictness row. A request with `wait-for-done`, no wants and no `done` (what `git fetch --negotiate-only` sends for `git push` with `push.negotiate`) is answered with `ACK` for every have that has a `commit_index` row, whatever version the row names, `NAK` when none has, and never `ready` (`upload_pack::fetch::negotiation_acks`; `FetchResponse` refuses `ready` under `wait-for-done`). This differs from the B2 negotiation row on purpose: those ACKs steer a fetch the planner must serve from exact versions; these only tell a push what to leave out. (2) Boundary outside the dependencies. The client never says which ref it will push, so an acknowledged commit can be on a ref the push does not depend on: a branch created from another branch, an amended forced push (forced updates do not depend on the branch), a merge into trunk, a tag, a new branch at an existing commit, several refs at once. Receive-pack finds the pack's boundary commits (`aspen_pack::boundary`: a second read of the spooled pack that resolves only commits and tags, parents and targets the pack lacks), plus any tip it did not send, looks them up in `commit_index`, and adds those versions' closures to the packs every read searches (`Push::widen`: before completion when a thin base is outside the dependencies, after the scan when the connectivity check misses). The writes' dependencies do not change: each write's pack is then planned by the forced-trunk walk, generalised (`Push::fill`), and holds everything its tip reaches that its dependencies lack, copied from storage where the client left it out. Every invariant stays as it was (connected; exact when the dependencies are; trunk never depends on a branch). While the search is widened, the objects completion inserted are not taken as held (item 2's changed-path walk otherwise marks every completion base held): a base may then be another ref's object, so only the lookup against the dependencies decides; trusting them stored a stacked branch's version without a base it reached, and the next push built on it failed with `missing object`. What the writes copy counts against the push's own byte limit (`PushLimits::max_pack_bytes`, the caller's quota included); over it the push is refused with `UnpackError::LeftOutTooLarge`, whose message says to push with `push.negotiate=false`. Counted as `aspen_push_widened_total`. (3) Documented in the README quickstart and `site/src/using-git.md` \| Through the plain URL the 40th push of a 40-commit branch received 15,248 bytes against 750 for the first (`push_negotiate.rs`, git 2.39.5), because the advertisement shows only trunk (S2); on `main`, `push.negotiate` failed with `server does not support wait-for-done` and Git fell back to the same resend. With this change the 40 pushes received 489 to 507 bytes each, stored 755 to 760 bytes each and took 42 to 54 ms each (43 ms median early and late, against 35 to 52 ms and growing without it); no push of that loop widened. Acknowledging only trunk's commits is safe without (2) but saves nothing for a branch; acknowledging branch commits without (2) failed seven ordinary push shapes with `missing delta base` or `missing object` (all pinned by `negotiated_pushes_built_on_other_refs_land_connected`, which also checks every ref connected, trunk exact after the merge, the merge push received at most two commits' worth, and `fsck --strict` on a fresh fetch of every ref). Copying instead of adding a dependency keeps one mechanism for every push shape, at the cost of server reads and of storing the copied objects again on those pushes only; a single-branch push could depend on the boundary's version instead (follow-up). The copy limit keeps a small push naming a large branch from making the server read and store more than the client could have sent (`copying_left_out_history_is_bounded_by_the_push_limit`). Resent objects without `push.negotiate` were already not stored again (#231; `resent_history_is_not_stored_again`: stored bytes flat at 755 to 759 while received bytes grow 20x) \|
1265	1265	\| 2026-11-06 \| GC sweep incremental and leased (throughput item 7). Replaces the C1 row's "no leader election is needed" for garbage collection, and its sweep's candidate query. (1) Incremental sweep. `MetaStore::sweep_orphan_packs(lease, grace, limit)` returns a `SweepBatch` and reads at most `2 × limit` rows: the walk, `pack_touches` rows after a watermark `(touched_at_us, pack_id)` whose grace is over, in that order (new index `pack_touches_due`, migration 0009), and the audit, the next `limit` packs by ID (wrapping), which stamps packs with no touch (legacy, as before) and re-offers old orphans the watermark passed. Candidates go through the unchanged lock and re-checking `DELETE`s, so safety does not depend on the cursor. The watermark (table `gc_cursors`) never passes a walked candidate the lock skipped (a transaction holds it and may roll back). Liveness rests on a rule now written down in SCHEMA.md: whatever stops naming a pack touches it in the same transaction (both shallow evictions already do), so its registration moves ahead of the watermark; the audit is the backstop. (2) One process at a time. Table `leases` and `MetaStore::acquire_lease(name, owner, ttl)`; `Compactor::collect_garbage` takes `GC_LEASE` under `CompactConfig::gc_owner` (`aspen serve`: the worker ID; default unique per process) or does nothing (`GcReport::lease_held_elsewhere`); every sweep transaction first extends the lease, fenced on owner and epoch, and is refused (`StoreError::lease_lost`) once it expired or changed hands, so a stalled holder cannot sweep after a takeover. A dead holder's lease expires `gc_lease` after its last extension. The pass runs where `ASPEN_ROLE` runs the background worker (item 3's `compact_task`). (3) Off the push path's budget. On Postgres the pass uses its own pool (`ASPEN_GC_DB_MAX_CONNECTIONS`, 1) with its own `statement_timeout` (`ASPEN_GC_DB_STATEMENT_TIMEOUT_SECS`, 10 s); SQLite shares the store (one writer anyway). (4) Settings `ASPEN_GC_EVERY_SECS` (300), `ASPEN_GC_SWEEP_BATCH` (1000; was a fixed 100 per sweep), `ASPEN_GC_LEASE_SECS` (300, at least 10); metrics `aspen_gc_passes_total{result}`, `aspen_gc_lease_held`, `aspen_gc_sweep_{batches,rows_examined,packs_deleted}_total`, histograms `aspen_gc_sweep_batch_duration_seconds` and `aspen_gc_sweep_batch_rows` \| the maintainers' AWS load test: the old candidate query (`packs` LEFT JOIN `pack_touches` with four NOT EXISTS, ORDER BY id LIMIT 100) took 7.9 s per run at 1.5 M packs, on every instance at startup and every 5 minutes, uncoordinated, on the push path's pool; it cost a throughput dip, a 5 s p99 and 52 "internal storage error; retry" in 246 k pushes. Once orphans are rare it reads every pack. Measured by `crates/aspen-meta/tests/sweep_cost.rs` (1 M packs, every 1,000th an orphan, debug build, one 4-vCPU machine, local Postgres 16): old query 1.0 s per pass with no orphan due (144 ms with orphans due); new caught-up batch 7.8 ms reading 1,213 rows; the one-time catch-up over 1 M registrations took 1,023 batches (slowest 150 ms) while 4 writers committed 34,252 pushes with no error. SQLite: 366 ms against 6.0 ms. Required CI runs 20,000 packs on both dialects and checks the bound and both scans' plans (index range scans, no sort, no merge or hash join; a join for the audit planned as a merge that read `pack_touches` from its start, so the audit looks the touch up per row). Rejected: an ID-cursor scan of every pack in bounded batches (bounds each transaction but each cycle still reads every pack); a `gc` job kind in `jobs` as the lease (needs a recurring job and a CHECK change, and job attempts and dead-job parking do not fit). Tests: store (both dialects) `the_sweep_resumes_where_it_stopped_in_bounded_batches`, `a_pack_unnamed_behind_the_cursor_survives_its_grace_then_goes`, `the_walk_never_passes_a_pack_an_open_transaction_holds` (Postgres; fails when the watermark passes held rows), `the_audit_stamps_a_legacy_pack_behind_the_walk`, `a_lease_has_one_holder_and_fences_a_stale_one`; `gc_race.rs` `a_crashed_workers_deletions_are_finished_by_the_next_pass` now also checks the dead process's lease blocks others until it expires; `aspen-cli` `one_instance_at_a_time_collects_garbage` (two `aspen serve` processes, metrics show one collects, the other takes over when it stops); every earlier GC and GC-race test still passes with its call sites given the lease; mutations G12 to G14 in `crates/aspen-compact/scripts/check-mutations.py` \|
1266		−\| 2026-11-07 \| Throughput 6a: repository and token rows are cached in memory for a short TTL. `Access::repo` keeps each repository row it reads for `AccessCaches::repo_ttl`, and `access::tokens::CachedTokens` (what `aspen serve` and `aspen-place` wrap `SqlTokens` in) keeps each token row for `token_ttl`; both default to 5 s (`ASPEN_REPO_CACHE_TTL_MS`, `ASPEN_TOKEN_CACHE_TTL_MS`, `0` to `60000`, `0` turns a cache off). Only rows that exist are kept: a miss is never remembered, so a repository or token created by another process works on its next request, and a caller cannot fill the cache with names that do not exist (it is also capped at 65,536 rows). Expiry counts from when the lookup began, so a token revoked elsewhere (`aspen token revoke`, another instance) is refused everywhere within the TTL of its revocation; a revoke through this instance's `CachedTokens::revoke` (or `forget`), and a repository change followed by `Access::forget_repo`, apply on the next request, and a lookup already in flight when they run cannot put the old row back (the cache's generation check). Token expiry needs no lookup: `AuthService` checks the cached record's `expires_at` against the clock on every request. SSH keys are not cached (one lookup per connection and re-check, issue #243). No cross-instance invalidation signal: Aspen has none today (no `LISTEN`/`NOTIFY`), and a 5 s bound costs one lookup per caller per instance per 5 s. Measured by `tests/access_statements.rs` (statements counted from sqlx's per-statement events): about 2.1 access-layer statements per HTTP request on `main` before, 0 on a warm cache after \| Every Git request read its token and its repository: with 3 servers at ~445 branch pushes/s and Postgres at 99.5% CPU, those were about 4 of the ~30 statements per push (two HTTP requests each). Neither row changes on the push path; repository settings have no in-process API yet (only `create_repo`), and revocation is an operator action for which seconds of delay are acceptable \|
1267		−\| 2026-11-07 \| Monorepo benchmark item 8, `.idx` cold start: (1) concurrent misses on one pack's `.idx` share one load (single flight per pack in `aspen_pack::locate::StoredPacks`; a leader dropped mid-read hands the load to a waiter); (2) whether a missed `.idx` is kept whole depends on its search set: in probe order, whole only while the set's whole indexes (file plus 8 bytes per object of offsets) fit `ASPEN_IDX_CACHE_BYTES` beside what the set already holds, the rest sampled (`crate::sampled`), so a set never evicts itself; (3) `ASPEN_IDX_CACHE_BYTES` defaults to 1 GiB (was 256 MiB), a bound and not a reservation; (4) `ASPEN_MEMORY_HIGH_WATER_BYTES` (default 80% of the cgroup limit, or of the machine's memory; `0` off): while resident memory is over it, admission refuses new fetches, pushes and stitches at once with `503` and `Retry-After` (`Busy::Memory`; a retryable `ERR` over SSH), and work already running finishes. Replaces the deferral in the A2 `.idx` cache row: one call no longer holds more than the budget, because the plan keeps the set within it. Metrics: `aspen_push_idx_loads_total`, `aspen_push_idx_sampled_loads_total`, `aspen_push_idx_load_waits_total`, `aspen_push_idx_read_bytes_total`, `aspen_admission_memory_refusals_total` \| The benchmark (11.6 M objects, 301 MB of `.idx`, 1x c7g.2xlarge) started 128 pushes at once: each missed and read every `.idx` itself, and the indexes (about 400 MiB kept whole) did not fit the 256 MiB cache, so every push reloaded them; the server reached 15.6 GB and was killed by the kernel, which loses every request in flight. Test `crates/aspen-pack/tests/idx_cold_start.rs` (synthetic `.idx` files of the benchmark's shape, 128 concurrent cold lookups twice, peak resident memory): before, 1,280 `.idx` loads for 7 packs at 1/100 scale, and at full scale the run hit its 1,147 MiB memory cap (128 copies of 46 MB indexes; 224 s and 1,280 loads with reads not held concurrently); after, 7 loads, peak +252 MiB with a 256 MiB cache (3 sampled) and +396 MiB with the 1 GiB default (none sampled), 0.6 s. Required CI runs the 1/100 shape and the unit tests (`concurrent_misses_on_a_pack_share_one_load`, `a_search_set_over_the_budget_is_part_sampled_and_never_reloaded`, `memory_pressure_refuses_new_work_until_it_eases`, `serve_refuses_new_work_under_memory_pressure_and_stays_up`); nightly runs the full-scale one. Sampling every `.idx` on demand (fanout and ranges only, no whole read) was weighed: a binary search by range reads costs about 13 round trips per lookup, against one or two for a sample, which reads the file once, streamed in bounded sections. Admission refusal reads `VmRSS`, so freed memory the allocator keeps counts until it is returned; the fetch path's stitch plan still reads every segment's `.idx` whole (not changed here) \|
1268		−\| 2026-11-07 \| ls-refs snapshots (throughput item 6c; `aspen_server::upload_pack::ref_snapshot`, `ASPEN_LS_REFS_CACHE_REFS`, default 250,000 refs per instance, `0` off). Stock `git ls-remote <url> <ref>` sends no `ref-prefix`, so each poll listed every ref from the database (768 rows in the load test). Each instance now keeps per repository the refs it last read plus the store's cursor, and each `ls-refs` asks `MetaStore::ref_changes` for the refs whose current version that cursor had not seen: one statement that reads no ref rows when nothing moved. Migration 0010 adds `ref_versions.txid` (Postgres: the writing transaction's ID, cursor = the statement's snapshot `xmax` plus its in-progress list, at most 256 entries, else `xmin`; SQLite: the version ID, cursor = max ID + 1). Staleness bound: zero. A request is answered only from a read that began after it arrived (its own, or a concurrent request's that started later), so a push acknowledged by any instance is in every listing requested after the acknowledgement. Over-budget repositories are listed from the store as before, retried after 60 s; least recently listed repositories are evicted first. Considered and rejected: a per-repository version counter (`repos.ref_version`), which would serialize every push to a repository on one row (the push path's contention is per ref, PLAN section 1), and a time-bounded cache, which breaks read-after-write across instances \| Measured in `crates/aspen-server/tests/ls_refs_snapshot.rs` on 768 refs: 20 polls cost 40 statements and 15,380 ref rows uncached, 20 statements and 0 ref rows cached. A cursor of "largest ID seen" alone would miss a Postgres push that commits after a later-numbered one (`ref_changes_find_a_write_that_commits_after_a_later_one`, which fails with the in-progress list removed) \|
1269		−\| 2026-11-07 \| Throughput item 6b: a branch push sends PostgreSQL at most 5 statements (4 in the steady state: the advertisement 1, the push 3), from 19 (21 with the two repository lookups, which item 6a caches). (1) Reads. The push reads trunk, the ref it builds on and every command's ref in one statement (`MetaStore::current_versions`), and the commit check uses those versions instead of reading each ref again after ingest; a verdict on a version that moved since loses its compare-and-swap and is recomputed against the new one (push path, step 4), so this changes cost, not outcomes; it makes losing a race on a hot trunk a little more likely. The search set's closure walk reads a chain of versions it has not seen in one recursive statement that stops at the roots it remembers (`MetaStore::reachable_versions`, at most 512 rows; behind three servers it was one read per version, about 15 per push), and a committed version's closure is remembered when its dependencies' are, so the next push to that branch on the same instance reads none. The advertisement reads trunk and its recent versions (item 10's dense window) in one statement (`MetaStore::current_and_recent`), only those above the newest it remembers, and takes item 10's other `.have` versions from the closure cache (`Closures::versions_at`), reading only those this instance lacks. A pack a push stored is put into the `.idx` cache (item 11's `StoredPacks::remember_pack`, which replaced this item's `remember_idx`), so the next push to that ref neither reads its row nor its file. (2) Writes on PostgreSQL. A push of one ref commits in one statement with no transaction: it takes an ID from the version table's identity sequence, moves the `refs` row to it with compare-and-swap (only if the pack, locked `FOR KEY SHARE`, still exists), inserts the version from the moved row, and writes its commit rows (up to 100) and jobs, all or nothing; foreign keys are checked at the end of the statement, so `refs_version_fk` sees the version it inserted. A lost race now writes nothing without an error (the `UPDATE` re-reads the row after the winner commits). An atomic multi-ref push keeps the transaction (its commit rows need one global order). A pack registration is one statement too: it inserts at generation 0 only when the statement's snapshot holds neither a row nor a tombstone for the checksum, and otherwise locks and touches the existing row; anything else (a tombstone, a row swept or inserted concurrently) falls back to the transaction, which still reads the tombstones after its insert. This relies on a sweep deleting only rows whose last registration is a grace period old, so they are in the snapshot: true while servers' clocks agree within `gc_grace`, which pushes already need. SQLite keeps every write in its transaction (no data-modifying `WITH`). (3) Group commit (replaces "every push (or pack) runs behind its own savepoint" in the group commit row): on PostgreSQL a push or pack that is one statement runs in the batch's transaction without a savepoint, so a batch of N pushes is N statements plus `BEGIN` and `COMMIT`; if any statement fails, the transaction is rolled back and the batch runs again with a savepoint around every push, so a failure still refuses only its own push. (4) Counting: every store statement runs on `aspen_meta::conn::Conn`, which counts it (`SqlStore::statements`, `MetaStore::count_statements`), reported per push in `Timings::db_queries` and `aspen_push_db_statements_total{kind}`. No index was missing: auto_explain on the push statements shows index scans only, and the recursive read uses `ref_versions_pkey` \| Item 6 of the push-throughput work: with three servers, branch pushes plateaued at about 445/s with PostgreSQL (db.m7g.xlarge) at 99.5% CPU and the servers at 25%, about 30 statements and 7 ms of database CPU per push. Local measurement (PostgreSQL 17, `pg_stat_statements` with `track_utility`, 4 writers each pushing 50 times to its own branch through stock git, one server, a 2-vCPU machine shared with the clients): on `6f74e3e` 21.28 statements per push (2 of them repository lookups, 4 `BEGIN`/`COMMIT`), 1.06 ms of statement execution and 2.5 ms of PostgreSQL CPU per push; after, 6.23 statements (2 of them repository lookups, none `BEGIN`/`COMMIT`), 0.80 ms and 2.0 ms. One server cannot show the closure walk's reads behind three; a 3,000-version chain is read in one 2.2 ms statement of 513 rows. Tests: `crates/aspen-server/tests/push_statements.rs` (PostgreSQL, stock git: each of 19 steady-state branch pushes sends at most 5 statements, only the repository lookups are outside receive-pack's count, and with item 6a's cache they average at most 5 with those lookups counted, 4 in practice; 8 concurrent pushes with group commit send at most 5 queries each, lookups included, and `BEGIN`/`COMMIT` only per batch), `crates/aspen-meta/tests/store.rs` (both dialects: `current_versions_reads_many_refs_in_one_statement`, `current_and_recent_reads_a_window_after_a_seq`, `reachable_versions_walks_dependencies_to_the_stops`, `a_branch_push_commits_in_few_statements`; PostgreSQL: `a_registration_that_waits_for_a_sweep_takes_the_next_generation`, which gets generation 0 instead of 1 when the snapshot check is removed). The store suite's compare-and-swap, swept-pack and group-commit isolation cases run unchanged against the new statements \|
1270		−\| 2026-11-07 \| Throughput item 10: the push advertisement's `.have` tips reach all of trunk's searchable history. With `n = have_tips` (stored per repository, default 8, 0 still advertises none) the tips are every trunk version from 1 to `8n` back, then distances each about a quarter more than the one before (`back + max(1, back / 4)`), then the oldest trunk version in the current trunk version's closure (`aspen_core::meta::have_tip_seqs`); a distance that falls in a gap of the closure (versions a forced trunk update skipped) takes the newest held version below it. Which versions are held comes from the push's own cached closure summary (`receive::closure::Closures::held_seqs`), not from reading each version back to check its trunk dependency, so the advertisement reads the versions pushed since the cached summary (one between two pushes on one instance) and one `versions_by_seq` batch, and the push that follows reuses the summary. Size: at most `8n + 2 + log_{5/4}(current / 8n)` lines (108 for `n = 8` and a million versions; 4 KB). A version is advertised only if it is in the closure the push searches, the condition of the B1 row "the push advertisement's `.have` tips" (that row's walk, which stopped at the first version whose trunk dependency is not its parent, is replaced by closure membership: a version a dependency path reaches is in the closure whichever dependency carries it). Replaces "up to 8 earlier trunk tips" (S2 row) and the log-spaced schedule (N1 row) \| 8 log-spaced tips reach 128 versions back, about 13 s at the monorepo benchmark's 9.6 pushes/s; any client further behind holds none of them, and stock `git push` without `push.negotiate` then sends its whole history (10.9 M objects for one commit in that benchmark). A window plus a 5/4 tail bounds what a client `d` behind resends to its own objects for `d <= 8n` and about `d / 4` landings beyond, while the advertisement grows only with the logarithm of trunk's length. Measured by `crates/aspen-server/tests/stale_base_push.rs` (stock git 2.39, `push.negotiate=false`, one-file landings 8 directories deep, objects from git's `Total` line). Before (`be8e85d`): 4,608 files and 150 landings, clients 5, 110 and 140 behind sent 44, 209 and 5,241 objects for an 11-object commit; 301,806 files and 400 landings, 5 and 300 behind sent 44 and 312,759. After: 11, 176 and 121; 11 and 44; advertisement 68 and 73 lines (3.7 and 3.9 KB, against 8 lines before). Required CI runs the 4,608-file case (about 11 s); nightly runs the 301,806-file one (about 100 s). Not measured here: real-deployment push throughput and p99, which the load test re-measures \|
	1266	+\| 2026-11-06 \| Throughput 6a: repository and token rows are cached in memory for a short TTL. `Access::repo` keeps each repository row it reads for `AccessCaches::repo_ttl`, and `access::tokens::CachedTokens` (what `aspen serve` and `aspen-place` wrap `SqlTokens` in) keeps each token row for `token_ttl`; both default to 5 s (`ASPEN_REPO_CACHE_TTL_MS`, `ASPEN_TOKEN_CACHE_TTL_MS`, `0` to `60000`, `0` turns a cache off). Only rows that exist are kept: a miss is never remembered, so a repository or token created by another process works on its next request, and a caller cannot fill the cache with names that do not exist (it is also capped at 65,536 rows). Expiry counts from when the lookup began, so a token revoked elsewhere (`aspen token revoke`, another instance) is refused everywhere within the TTL of its revocation; a revoke through this instance's `CachedTokens::revoke` (or `forget`), and a repository change followed by `Access::forget_repo`, apply on the next request, and a lookup already in flight when they run cannot put the old row back (the cache's generation check). Token expiry needs no lookup: `AuthService` checks the cached record's `expires_at` against the clock on every request. SSH keys are not cached (one lookup per connection and re-check, issue #243). No cross-instance invalidation signal: Aspen has none today (no `LISTEN`/`NOTIFY`), and a 5 s bound costs one lookup per caller per instance per 5 s. Measured by `tests/access_statements.rs` (statements counted from sqlx's per-statement events): about 2.1 access-layer statements per HTTP request on `main` before, 0 on a warm cache after \| Every Git request read its token and its repository: with 3 servers at ~445 branch pushes/s and Postgres at 99.5% CPU, those were about 4 of the ~30 statements per push (two HTTP requests each). Neither row changes on the push path; repository settings have no in-process API yet (only `create_repo`), and revocation is an operator action for which seconds of delay are acceptable \|
	1267	+\| 2026-11-06 \| Monorepo benchmark item 8, `.idx` cold start: (1) concurrent misses on one pack's `.idx` share one load (single flight per pack in `aspen_pack::locate::StoredPacks`; a leader dropped mid-read hands the load to a waiter); (2) whether a missed `.idx` is kept whole depends on its search set: in probe order, whole only while the set's whole indexes (file plus 8 bytes per object of offsets) fit `ASPEN_IDX_CACHE_BYTES` beside what the set already holds, the rest sampled (`crate::sampled`), so a set never evicts itself; (3) `ASPEN_IDX_CACHE_BYTES` defaults to 1 GiB (was 256 MiB), a bound and not a reservation; (4) `ASPEN_MEMORY_HIGH_WATER_BYTES` (default 80% of the cgroup limit, or of the machine's memory; `0` off): while resident memory is over it, admission refuses new fetches, pushes and stitches at once with `503` and `Retry-After` (`Busy::Memory`; a retryable `ERR` over SSH), and work already running finishes. Replaces the deferral in the A2 `.idx` cache row: one call no longer holds more than the budget, because the plan keeps the set within it. Metrics: `aspen_push_idx_loads_total`, `aspen_push_idx_sampled_loads_total`, `aspen_push_idx_load_waits_total`, `aspen_push_idx_read_bytes_total`, `aspen_admission_memory_refusals_total` \| The benchmark (11.6 M objects, 301 MB of `.idx`, 1x c7g.2xlarge) started 128 pushes at once: each missed and read every `.idx` itself, and the indexes (about 400 MiB kept whole) did not fit the 256 MiB cache, so every push reloaded them; the server reached 15.6 GB and was killed by the kernel, which loses every request in flight. Test `crates/aspen-pack/tests/idx_cold_start.rs` (synthetic `.idx` files of the benchmark's shape, 128 concurrent cold lookups twice, peak resident memory): before, 1,280 `.idx` loads for 7 packs at 1/100 scale, and at full scale the run hit its 1,147 MiB memory cap (128 copies of 46 MB indexes; 224 s and 1,280 loads with reads not held concurrently); after, 7 loads, peak +252 MiB with a 256 MiB cache (3 sampled) and +396 MiB with the 1 GiB default (none sampled), 0.6 s. Required CI runs the 1/100 shape and the unit tests (`concurrent_misses_on_a_pack_share_one_load`, `a_search_set_over_the_budget_is_part_sampled_and_never_reloaded`, `memory_pressure_refuses_new_work_until_it_eases`, `serve_refuses_new_work_under_memory_pressure_and_stays_up`); nightly runs the full-scale one. Sampling every `.idx` on demand (fanout and ranges only, no whole read) was weighed: a binary search by range reads costs about 13 round trips per lookup, against one or two for a sample, which reads the file once, streamed in bounded sections. Admission refusal reads `VmRSS`, so freed memory the allocator keeps counts until it is returned; the fetch path's stitch plan still reads every segment's `.idx` whole (not changed here) \|
	1268	+\| 2026-11-06 \| ls-refs snapshots (throughput item 6c; `aspen_server::upload_pack::ref_snapshot`, `ASPEN_LS_REFS_CACHE_REFS`, default 250,000 refs per instance, `0` off). Stock `git ls-remote <url> <ref>` sends no `ref-prefix`, so each poll listed every ref from the database (768 rows in the load test). Each instance now keeps per repository the refs it last read plus the store's cursor, and each `ls-refs` asks `MetaStore::ref_changes` for the refs whose current version that cursor had not seen: one statement that reads no ref rows when nothing moved. Migration 0010 adds `ref_versions.txid` (Postgres: the writing transaction's ID, cursor = the statement's snapshot `xmax` plus its in-progress list, at most 256 entries, else `xmin`; SQLite: the version ID, cursor = max ID + 1). Staleness bound: zero. A request is answered only from a read that began after it arrived (its own, or a concurrent request's that started later), so a push acknowledged by any instance is in every listing requested after the acknowledgement. Over-budget repositories are listed from the store as before, retried after 60 s; least recently listed repositories are evicted first. Considered and rejected: a per-repository version counter (`repos.ref_version`), which would serialize every push to a repository on one row (the push path's contention is per ref, PLAN section 1), and a time-bounded cache, which breaks read-after-write across instances \| Measured in `crates/aspen-server/tests/ls_refs_snapshot.rs` on 768 refs: 20 polls cost 40 statements and 15,380 ref rows uncached, 20 statements and 0 ref rows cached. A cursor of "largest ID seen" alone would miss a Postgres push that commits after a later-numbered one (`ref_changes_find_a_write_that_commits_after_a_later_one`, which fails with the in-progress list removed) \|
	1269	+\| 2026-11-06 \| Throughput item 6b: a branch push sends PostgreSQL at most 5 statements (4 in the steady state: the advertisement 1, the push 3), from 19 (21 with the two repository lookups, which item 6a caches). (1) Reads. The push reads trunk, the ref it builds on and every command's ref in one statement (`MetaStore::current_versions`), and the commit check uses those versions instead of reading each ref again after ingest; a verdict on a version that moved since loses its compare-and-swap and is recomputed against the new one (push path, step 4), so this changes cost, not outcomes; it makes losing a race on a hot trunk a little more likely. The search set's closure walk reads a chain of versions it has not seen in one recursive statement that stops at the roots it remembers (`MetaStore::reachable_versions`, at most 512 rows; behind three servers it was one read per version, about 15 per push), and a committed version's closure is remembered when its dependencies' are, so the next push to that branch on the same instance reads none. The advertisement reads trunk and its recent versions (item 10's dense window) in one statement (`MetaStore::current_and_recent`), only those above the newest it remembers, and takes item 10's other `.have` versions from the closure cache (`Closures::versions_at`), reading only those this instance lacks. A pack a push stored is put into the `.idx` cache (item 11's `StoredPacks::remember_pack`, which replaced this item's `remember_idx`), so the next push to that ref neither reads its row nor its file. (2) Writes on PostgreSQL. A push of one ref commits in one statement with no transaction: it takes an ID from the version table's identity sequence, moves the `refs` row to it with compare-and-swap (only if the pack, locked `FOR KEY SHARE`, still exists), inserts the version from the moved row, and writes its commit rows (up to 100) and jobs, all or nothing; foreign keys are checked at the end of the statement, so `refs_version_fk` sees the version it inserted. A lost race now writes nothing without an error (the `UPDATE` re-reads the row after the winner commits). An atomic multi-ref push keeps the transaction (its commit rows need one global order). A pack registration is one statement too: it inserts at generation 0 only when the statement's snapshot holds neither a row nor a tombstone for the checksum, and otherwise locks and touches the existing row; anything else (a tombstone, a row swept or inserted concurrently) falls back to the transaction, which still reads the tombstones after its insert. This relies on a sweep deleting only rows whose last registration is a grace period old, so they are in the snapshot: true while servers' clocks agree within `gc_grace`, which pushes already need. SQLite keeps every write in its transaction (no data-modifying `WITH`). (3) Group commit (replaces "every push (or pack) runs behind its own savepoint" in the group commit row): on PostgreSQL a push or pack that is one statement runs in the batch's transaction without a savepoint, so a batch of N pushes is N statements plus `BEGIN` and `COMMIT`; if any statement fails, the transaction is rolled back and the batch runs again with a savepoint around every push, so a failure still refuses only its own push. (4) Counting: every store statement runs on `aspen_meta::conn::Conn`, which counts it (`SqlStore::statements`, `MetaStore::count_statements`), reported per push in `Timings::db_queries` and `aspen_push_db_statements_total{kind}`. No index was missing: auto_explain on the push statements shows index scans only, and the recursive read uses `ref_versions_pkey` \| Item 6 of the push-throughput work: with three servers, branch pushes plateaued at about 445/s with PostgreSQL (db.m7g.xlarge) at 99.5% CPU and the servers at 25%, about 30 statements and 7 ms of database CPU per push. Local measurement (PostgreSQL 17, `pg_stat_statements` with `track_utility`, 4 writers each pushing 50 times to its own branch through stock git, one server, a 2-vCPU machine shared with the clients): on `6f74e3e` 21.28 statements per push (2 of them repository lookups, 4 `BEGIN`/`COMMIT`), 1.06 ms of statement execution and 2.5 ms of PostgreSQL CPU per push; after, 6.23 statements (2 of them repository lookups, none `BEGIN`/`COMMIT`), 0.80 ms and 2.0 ms. One server cannot show the closure walk's reads behind three; a 3,000-version chain is read in one 2.2 ms statement of 513 rows. Tests: `crates/aspen-server/tests/push_statements.rs` (PostgreSQL, stock git: each of 19 steady-state branch pushes sends at most 5 statements, only the repository lookups are outside receive-pack's count, and with item 6a's cache they average at most 5 with those lookups counted, 4 in practice; 8 concurrent pushes with group commit send at most 5 queries each, lookups included, and `BEGIN`/`COMMIT` only per batch), `crates/aspen-meta/tests/store.rs` (both dialects: `current_versions_reads_many_refs_in_one_statement`, `current_and_recent_reads_a_window_after_a_seq`, `reachable_versions_walks_dependencies_to_the_stops`, `a_branch_push_commits_in_few_statements`; PostgreSQL: `a_registration_that_waits_for_a_sweep_takes_the_next_generation`, which gets generation 0 instead of 1 when the snapshot check is removed). The store suite's compare-and-swap, swept-pack and group-commit isolation cases run unchanged against the new statements \|
	1270	+\| 2026-11-06 \| Throughput item 10: the push advertisement's `.have` tips reach all of trunk's searchable history. With `n = have_tips` (stored per repository, default 8, 0 still advertises none) the tips are every trunk version from 1 to `8n` back, then distances each about a quarter more than the one before (`back + max(1, back / 4)`), then the oldest trunk version in the current trunk version's closure (`aspen_core::meta::have_tip_seqs`); a distance that falls in a gap of the closure (versions a forced trunk update skipped) takes the newest held version below it. Which versions are held comes from the push's own cached closure summary (`receive::closure::Closures::held_seqs`), not from reading each version back to check its trunk dependency, so the advertisement reads the versions pushed since the cached summary (one between two pushes on one instance) and one `versions_by_seq` batch, and the push that follows reuses the summary. Size: at most `8n + 2 + log_{5/4}(current / 8n)` lines (108 for `n = 8` and a million versions; 4 KB). A version is advertised only if it is in the closure the push searches, the condition of the B1 row "the push advertisement's `.have` tips" (that row's walk, which stopped at the first version whose trunk dependency is not its parent, is replaced by closure membership: a version a dependency path reaches is in the closure whichever dependency carries it). Replaces "up to 8 earlier trunk tips" (S2 row) and the log-spaced schedule (N1 row) \| 8 log-spaced tips reach 128 versions back, about 13 s at the monorepo benchmark's 9.6 pushes/s; any client further behind holds none of them, and stock `git push` without `push.negotiate` then sends its whole history (10.9 M objects for one commit in that benchmark). A window plus a 5/4 tail bounds what a client `d` behind resends to its own objects for `d <= 8n` and about `d / 4` landings beyond, while the advertisement grows only with the logarithm of trunk's length. Measured by `crates/aspen-server/tests/stale_base_push.rs` (stock git 2.39, `push.negotiate=false`, one-file landings 8 directories deep, objects from git's `Total` line). Before (`be8e85d`): 4,608 files and 150 landings, clients 5, 110 and 140 behind sent 44, 209 and 5,241 objects for an 11-object commit; 301,806 files and 400 landings, 5 and 300 behind sent 44 and 312,759. After: 11, 176 and 121; 11 and 44; advertisement 68 and 73 lines (3.7 and 3.9 KB, against 8 lines before). Required CI runs the 4,608-file case (about 11 s); nightly runs the 301,806-file one (about 100 s). Not measured here: real-deployment push throughput and p99, which the load test re-measures \|
1271	1271	\| 2026-11-06 \| Merge queue track A: one builder per repository, `git push -o land`, a batch interval. (1) A worker builds a repository's batches only while it holds the lease `queue:<repo id>` (`MetaStore::acquire_lease`, the fenced lease of migration 0009), renewed before each batch and lapsing `QueueConfig::lease_ttl` (10 s) after its last renewal; the holder polls the repository every interval (a queue push wakes only the instance that took it), the others poll to take over a lapsed lease, and an idle holder lets its lease lapse at once. (2) `git push -o land` of a branch writes `refs/queue/<trunk>/<branch>` at the branch's new version value and dependencies in the branch's own `commit_push` transaction (a compare-and-swap on the queue ref's current version); refused with the queue off. (3) `ASPEN_MERGE_QUEUE_INTERVAL_MS` (default 50, at most 10000): a batch that is not full waits until its oldest entry is that old \| Without the lease every instance built the same batch and all but one lost the compare-and-swap, wasting reads and pushes; exactly-once still rests only on the atomic compare-and-swap push (a holder that stalled past its expiry lands through it or not at all), so the lease is an efficiency measure, not a correctness one. Writing the queue ref beside the branch reuses the branch's pack and dependencies, so the queue version is connected by construction and enqueueing costs no second push. The interval trades latency for batch size. Tests: `one_worker_builds_at_a_time_and_a_dead_holders_lease_is_taken_over`, `push_option_land_queues_the_branch`, `a_batch_waits_for_the_interval_unless_it_is_full`; `outcomes_older_than_the_ttl_are_deleted` now gives its two configurations one lease owner \|
1272		−\| 2026-11-07 \| Throughput item 11 (monorepo benchmark: 300,000 files, 11.6 M objects, S3; prefetch about 666 ms and graph about 489 ms per push, nearly all of it waiting on the store): a push's round trips to the store do not grow with delta-chain depth or with the repository. (1) Chain maps. `aspen_pack::locate` keeps, beside every whole `.idx`, each entry's base position from the pack's entry table (read with the `.idx`, 4 bytes per object), and beside a sampled `.idx` a chain map on the spool disk (`aspen_pack::chains`: 16 bytes per object in an unnamed file under `<spool>/chains`, one offset per 256 entries in memory; REF_DELTA bases, named by ID, are left to be found by reading). A prefetch, and the walk of the changed paths reading old trees through it, reads every entry of every chain in one round instead of one round per level; an entry table that does not describe its pack is corruption. (2) Lookups in sampled `.idx` files read a window's IDs and its offsets together (one round trip instead of two; a miss now reads the 1 KiB of offsets for nothing) and probe every sampled pack that comes before an ID's first whole-pack hit at once (one round trip however many large packs; the earliest in probe order still answers; a later sampled pack may be read for an ID an earlier one holds). (3) The previous push's `.idx`: the receive path hands each pack it stored, whole-sized, with its chain map to the locator (`StoredPacks::remember_pack`), so the next push to the ref does not load it (a round trip per push). (4) One lookup for held objects and connectivity: the objects the walk did not prove held and the references the connectivity check needs are looked up together, and the check looks up again only what that lookup did not find (when the search was not widened). (5) `ASPEN_PREFETCH_PARALLEL_READS` defaults to 256 (was 64), so the chains of a push's bases and its sampled lookups fit in one wave. (6) Compaction bounds trees at depth 10 (`CompactConfig::max_tree_delta_depth`, `RepackOptions::max_tree_delta_depth`; blobs keep 50): keep-first chains each version of a hot tree onto the one before, so compaction packs held the root and busy directories at depth 50; an entry's kind is its chain's, and `verify_pack` reports `max_tree_delta_depth` for the compactor's check. Measured (`crates/aspen-server/tests/push_round_trips.rs`, a blob store adding 50 ms per read, round trips counted by the store): a push onto 50-deep chains on a cold server 61 → 11 round trips (prefetch 53.5 → 2.2 rounds of wall time); per warm push, 65,610 files in one large pack 4.0 → 3.0 and in 8 large packs 18.0 → 3.0; 301,806 files (nightly) 4.2 → 3.0 and 18.0 → 3.0. Not done: a cold walk of the changed path still reads each uncached old tree in turn (one round trip a level, two in a sampled `.idx`), since each names the next. With item 8 (single-flight `.idx` loads, `plan_misses`), a whole `.idx` is costed at 12 bytes per object beside the file, not 8; a sampled one's chain map is on disk and costs the cache almost nothing. Real-deployment throughput is re-measured by the load test \| Prefetch learned each base's base only by reading it, and lookups probed large packs one after another, so per-push latency grew with chain depth and with the number of large packs while CPU stayed at 18%. A chain map in memory for a sampled pack would cost 12 bytes per object (139 MB at 11.6 M objects), so it lives on local disk; a separate depth for trees keeps the storage cost of a lower cap off blobs, which pushes need far less often as bases #319 folded in: its `remember_idx` (cache a just-stored pack's `.idx`) becomes `remember_pack`, which also builds the chain map from the uploaded entry table; an `.idx` cached without one would leave chain lookups without base positions. \|
	1272	+\| 2026-11-06 \| Throughput item 11 (monorepo benchmark: 300,000 files, 11.6 M objects, S3; prefetch about 666 ms and graph about 489 ms per push, nearly all of it waiting on the store): a push's round trips to the store do not grow with delta-chain depth or with the repository. (1) Chain maps. `aspen_pack::locate` keeps, beside every whole `.idx`, each entry's base position from the pack's entry table (read with the `.idx`, 4 bytes per object), and beside a sampled `.idx` a chain map on the spool disk (`aspen_pack::chains`: 16 bytes per object in an unnamed file under `<spool>/chains`, one offset per 256 entries in memory; REF_DELTA bases, named by ID, are left to be found by reading). A prefetch, and the walk of the changed paths reading old trees through it, reads every entry of every chain in one round instead of one round per level; an entry table that does not describe its pack is corruption. (2) Lookups in sampled `.idx` files read a window's IDs and its offsets together (one round trip instead of two; a miss now reads the 1 KiB of offsets for nothing) and probe every sampled pack that comes before an ID's first whole-pack hit at once (one round trip however many large packs; the earliest in probe order still answers; a later sampled pack may be read for an ID an earlier one holds). (3) The previous push's `.idx`: the receive path hands each pack it stored, whole-sized, with its chain map to the locator (`StoredPacks::remember_pack`), so the next push to the ref does not load it (a round trip per push). (4) One lookup for held objects and connectivity: the objects the walk did not prove held and the references the connectivity check needs are looked up together, and the check looks up again only what that lookup did not find (when the search was not widened). (5) `ASPEN_PREFETCH_PARALLEL_READS` defaults to 256 (was 64), so the chains of a push's bases and its sampled lookups fit in one wave. (6) Compaction bounds trees at depth 10 (`CompactConfig::max_tree_delta_depth`, `RepackOptions::max_tree_delta_depth`; blobs keep 50): keep-first chains each version of a hot tree onto the one before, so compaction packs held the root and busy directories at depth 50; an entry's kind is its chain's, and `verify_pack` reports `max_tree_delta_depth` for the compactor's check. Measured (`crates/aspen-server/tests/push_round_trips.rs`, a blob store adding 50 ms per read, round trips counted by the store): a push onto 50-deep chains on a cold server 61 → 11 round trips (prefetch 53.5 → 2.2 rounds of wall time); per warm push, 65,610 files in one large pack 4.0 → 3.0 and in 8 large packs 18.0 → 3.0; 301,806 files (nightly) 4.2 → 3.0 and 18.0 → 3.0. Not done: a cold walk of the changed path still reads each uncached old tree in turn (one round trip a level, two in a sampled `.idx`), since each names the next. With item 8 (single-flight `.idx` loads, `plan_misses`), a whole `.idx` is costed at 12 bytes per object beside the file, not 8; a sampled one's chain map is on disk and costs the cache almost nothing. Real-deployment throughput is re-measured by the load test \| Prefetch learned each base's base only by reading it, and lookups probed large packs one after another, so per-push latency grew with chain depth and with the number of large packs while CPU stayed at 18%. A chain map in memory for a sampled pack would cost 12 bytes per object (139 MB at 11.6 M objects), so it lives on local disk; a separate depth for trees keeps the storage cost of a lower cap off blobs, which pushes need far less often as bases #319 folded in: its `remember_idx` (cache a just-stored pack's `.idx`) becomes `remember_pack`, which also builds the chain map from the uploaded entry table; an `.idx` cached without one would leave chain lookups without base positions. \|
1273	1273	\| 2026-11-06 \| Layered shallow answers (monorepo item 9). A shallow fetch eligible for an exact pack whose version has none yet waits for the build up to `ShallowCacheConfig::layered_wait` (500 ms), then is served the version's layered answer if it has one (`aspen_server::upload_pack::layered`): one stitched pack, never packfile URIs, of a patch pack and a base. The base is the exact pack recorded for the newest earlier version of the same ref at the same depth (found from a per-process hint, else by probing up to 64 versions back; at most 256 back), with its own commits and tags left out by the stitcher (`StitchPlan::leaving_out`, found from its entry table). The patch holds every other object the region reaches (the new commits and tags, the trees and blobs changed since, and anything only older history holds), read from the packs pushed since or the version's closure (receive-pack's closure cache), written whole, registered and uploaded as a level-1 compaction pack no cover or recording names. Every object sent resolves its links: the base's trees and blobs are closed under reachability, and the only commits are the new region's. A first version sent the base whole plus the pushes' own packs; the chaos smoke's clients (`fetch.fsckObjects`) refused it ("did not receive expected object": the earlier commits' parents and the intermediate trees' entries were not sent). The patch is found by a walk that stops at any tree or blob of the base, so it reads only the commits and trees that changed since the base, one batch per tree level, under `max_layered_bytes` (256 MiB, also the patch's cap). Only a request with one want is served a layered answer (a left-out commit could be another want's). The answer is computed once per version and depth per process (reused 60 s) and is also the exact build's sources, which replaces #266's incremental sources (whose build fell back to the full closure on any missing object, such as a revert). A request whose build slot is busy or whose last build failed is served the layered answer too. Without one (a cold start: no recorded version within reach) the request waits up to `build_wait` (20 s) and falls back to the full closure as before. New metrics: `aspen_shallow_exact_fetches_total{pack="layered"}`, `aspen_shallow_layered_patch_objects_total`, `aspen_shallow_wait_seconds` \| The monorepo benchmark (300,000 files, 11.6 M objects): a new trunk tip's exact-pack build took longer than the 20 s wait, so depth-1 clones fell back to the full history (1.7 GB in 109 s, about 3 GB server memory each; 16 CI clones OOM-killed the server), and trunk moves on every landing. A build copies and verifies a whole snapshot, so it cannot fit a sub-second budget at that size; serving the previous snapshot plus the landings since can, and is at most one snapshot plus what changed. Measured on this 4-vCPU machine, local blob store, with `crates/aspen-server/tests/shallow_scale.rs` (300,000 files eight levels deep, 250 MB depth-1 pack, 750 MB of history, 16 concurrent `git fetch --depth=1` after one landing and after five landings plus a revert): on `main` one run sent 16 of 16 clones after the revert the full 750 MB history (16 `wait_timed_out` fallbacks, server heap +1.4 GB) and another kept every clone waiting for its build (slowest clone 44 s against 32.5 s once recorded); with layered answers the longest server wait was 0.70 s and 1.30 s, 0 fallbacks, every clone 250 MB, heap at most +736 MB for all 16 clones. Required CI: `a_new_tip_is_served_the_last_recorded_pack_and_the_pushes_since` and `a_revert_is_patched_into_the_layered_answer` (each fails with layered answers turned off: the fetch falls back to the full history; the second also fails without the patch: `fatal: missing blob object`; both fail when the base's commits are not left out: `fatal: did not receive expected object`; their clients fetch with `fetch.fsckObjects`), and `left_out_objects_are_dropped_from_every_segment` in aspen-pack. Nightly: `shallow-scale`. Not measured: against S3, where a build is slower and the answer's reads are round trips \|
1274	1274	\| 2026-11-06 \| Public-share sizing (place change 3). For a public share where people push the demo to its limit (about 500 pushes/s in total), the demo environment's place task is one 16 vCPU / 64 GiB Fargate task (Fargate's largest CPU size), the shared database is `db.m7g.2xlarge` (8 vCPU, 32 GiB) with 400 GiB gp3 (12,000 IOPS, 500 MiB/s), and aspen-place's pool is 64 connections (`PLACE_DB_MAX_CONNECTIONS`, new; default 10). Sizes are per environment in the environment root (`sizing` in `main.tf`, each overridable by a variable), so any apply of the demo from `main` keeps them; staging keeps 0.5 vCPU / 1 GiB and `db.t4g.micro`. Database changes apply at once (`db_apply_immediately`, a module variable, default false; true in the environment root). The account budget is $20,000/month (validation and `account-budget`'s `max_limit_usd` ceiling raised from $300) and a new daily budget `aspen-account-daily` alerts at 50/80/100% of $650; neither has a budget action, so nothing shuts down. Estimated fixed cost of the demo environment about $43/day; at a sustained 500 pushes/s, S3 PUTs (pack, `.idx` and entry table per push, about 1,500/s) add about $650/day \| the maintainers approved up to about $650/day for a week. The throughput bench reached 567 pushes/s with 8 cores for aspen-place and a 4-core PostgreSQL at about 85%, so both get about twice that. aspen-place keeps the canvas, merge queue and event feed in memory and must never run twice (`demo/place/server/README.md`, "Limits of this version"), so it scales up, not out. Below 400 GiB, RDS gp3 has 3,000 IOPS whatever the size; 400 GiB is the smallest size with 12,000. A budget at $300 would alert within hours of the share and say nothing useful; a daily budget shows a runaway day within about a day (AWS Budgets' billing data lags by up to a day) \|
1275		−\| 2026-11-07 \| Place change 7: `main`'s history is bounded by resetting it, not by deltas. Once the packs written on `main` since its root reach `PLACE_HISTORY_MAX_BYTES` (`PlaceConfig::history_max_bytes`, default 8 MiB; 0 never), the merger, after a merge, merges what is pending and force-updates `main` to one root commit holding the same canvas (`Place::bound_history`, the operator endpoint's `restart_history` without its branch deletion). Every server commit on `main` records the size so far in a `Place-History: root=<oid> commits=<n> bytes=<n>` trailer, so a restart keeps counting; a `main` with parents and no trailer counts as over the limit. The root commit names the old tip and carries the memory-of-recent-paint trailers. The tile hook, when a push's range reaches a root commit or the commit limit, excludes the commits of any version of `main` the push's history reaches first (`hook::MainHistory`, from the commit index: a commit is on `main`'s history exactly when its row names a version of `main`, which only the server writes; one batched `lookup_commits` per generation), so clones and branches from before a reset keep pushing and merging; ordinary pushes never take that path. The merger's interval adapts to load: `merge_interval` (3 s) up to `busy_pushes_per_second` (5, measured since the last merge), proportionally longer above it, at most `merge_interval_max` (30 s) (`place::merge_wait`). The canvas mirror sidecar now force-pushes `main`. Replaces "every merge stores a whole tree, and a full clone fetches all of them" (C7 rows) and the reset endpoint's "garbage collection then deletes the packs only the old history reached", which never held (ref versions are never deleted, so their packs stay) \| Deltas would not bound anything: every stored pack is self-contained (S3), so a merge pack deltified against the previous tree stores that tree again, compaction keeps entries as they are, and even deltified history grows with every merge. A size-triggered reset bounds a full clone by construction and needs no core change. Measured (`demo/place/server/tests/history.rs`, debug build, local store): 129 merges on a full 64 x 64 canvas give a 3,041 KiB full clone without the limit and 247 KiB with a 256 KiB limit (12 resets; `--depth=1` 25 KiB); the public demo's clone was 13 MiB. Tests: the clone bound and every stroke in the final canvas; a pending stroke merged into the root, a branch on the old history pushing and merging on, a `--depth=1` and a full clone two resets old pushing new branches, a non-tile file on old history and an unrelated root still refused, the old tip fetchable by the ID its root names, the count back after a restart; 224 concurrent strokes plus 16 stock-git pushes under the background loop with 3 or more resets, none lost; mutations P24 to P26 in `demo/place/server/scripts/check-mutations.py`. Load (ignored test, release, 64 writers, in-process strokes through receive-pack on SQLite, resets every few merges at a 64 KiB stress limit, on a shared 4-vCPU machine at load average 11 to 20): about 190 strokes/s sustained for 60 s, the machine's ceiling with or without resets (193/s with them off), 0 lost, p99 0.8 s against 0.6 s without resets. 500 pushes/s was not reached on this machine; the merger's work per run does not grow with the rate. Known cost: a branch built on an earlier history makes full clones fetch that history until it expires \|
	1275	+\| 2026-11-06 \| Place change 7: `main`'s history is bounded by resetting it, not by deltas. Once the packs written on `main` since its root reach `PLACE_HISTORY_MAX_BYTES` (`PlaceConfig::history_max_bytes`, default 8 MiB; 0 never), the merger, after a merge, merges what is pending and force-updates `main` to one root commit holding the same canvas (`Place::bound_history`, the operator endpoint's `restart_history` without its branch deletion). Every server commit on `main` records the size so far in a `Place-History: root=<oid> commits=<n> bytes=<n>` trailer, so a restart keeps counting; a `main` with parents and no trailer counts as over the limit. The root commit names the old tip and carries the memory-of-recent-paint trailers. The tile hook, when a push's range reaches a root commit or the commit limit, excludes the commits of any version of `main` the push's history reaches first (`hook::MainHistory`, from the commit index: a commit is on `main`'s history exactly when its row names a version of `main`, which only the server writes; one batched `lookup_commits` per generation), so clones and branches from before a reset keep pushing and merging; ordinary pushes never take that path. The merger's interval adapts to load: `merge_interval` (3 s) up to `busy_pushes_per_second` (5, measured since the last merge), proportionally longer above it, at most `merge_interval_max` (30 s) (`place::merge_wait`). The canvas mirror sidecar now force-pushes `main`. Replaces "every merge stores a whole tree, and a full clone fetches all of them" (C7 rows) and the reset endpoint's "garbage collection then deletes the packs only the old history reached", which never held (ref versions are never deleted, so their packs stay) \| Deltas would not bound anything: every stored pack is self-contained (S3), so a merge pack deltified against the previous tree stores that tree again, compaction keeps entries as they are, and even deltified history grows with every merge. A size-triggered reset bounds a full clone by construction and needs no core change. Measured (`demo/place/server/tests/history.rs`, debug build, local store): 129 merges on a full 64 x 64 canvas give a 3,041 KiB full clone without the limit and 247 KiB with a 256 KiB limit (12 resets; `--depth=1` 25 KiB); the public demo's clone was 13 MiB. Tests: the clone bound and every stroke in the final canvas; a pending stroke merged into the root, a branch on the old history pushing and merging on, a `--depth=1` and a full clone two resets old pushing new branches, a non-tile file on old history and an unrelated root still refused, the old tip fetchable by the ID its root names, the count back after a restart; 224 concurrent strokes plus 16 stock-git pushes under the background loop with 3 or more resets, none lost; mutations P24 to P26 in `demo/place/server/scripts/check-mutations.py`. Load (ignored test, release, 64 writers, in-process strokes through receive-pack on SQLite, resets every few merges at a 64 KiB stress limit, on a shared 4-vCPU machine at load average 11 to 20): about 190 strokes/s sustained for 60 s, the machine's ceiling with or without resets (193/s with them off), 0 lost, p99 0.8 s against 0.6 s without resets. 500 pushes/s was not reached on this machine; the merger's work per run does not grow with the rate. Known cost: a branch built on an earlier history makes full clones fetch that history until it expires \|
	1276	+\| 2026-11-06 \| Place change 7 review follow-up: the hook's walk of a push built on an earlier `main` is bounded and comes first. The range is walked `hook::PROBE_COMMITS` (64) deep first; only a push that reaches a root commit or goes deeper runs the boundary walk and a second range walk, so a clone from before a reset reads the same few dozen commits however long the old history is, instead of walking it down to its root (or to `max_commits`) first. The boundary walk (`hook::main_boundary`) spends at most `max_commits` units in all, one per ID asked of the commit index (in batches of at most 512) and one per object read, unreadable IDs and wide (octopus) generations included, and keeps no more queued IDs than it can still pay for; `MainHistory::classify` takes a `VersionCache` that lives for the whole walk, so each ref version is looked up once per walk. The canvas mirror sidecar pushes with `--force-with-lease` over the `main` it last pushed (read from GitHub at start), so a write by anyone else is reported, never overwritten. Mutation P26 now applies (its pattern predated rustfmt's line split) \| Independent review of #343: a stale clone paid a walk of the old history before the boundary was looked for, and the boundary walk's bound missed classified-but-unreadable IDs and whole generations. Tests (`hook::walk_tests`, counting every index lookup and object read against an in-memory history): a 5,000-parent octopus generation and a 10,000-commit chain off `main` stay within a 1,000 budget; a three-commit push on a 200- and a 5,000-commit old history costs the same (at most 80), and an ordinary push asks the index nothing. Mutations P26 to P29 each make their test fail. Checked in the stores: `commit_index` rows are written in the push transaction only after every ref write succeeded (`crates/aspen-meta/src/push.rs` `write_all`, rows from `PushCommit::rows`), so a refused update of `main` leaves no row naming a version of it; `Place::merge_once` takes `merge_lock` itself \|
	1277	+\| 2026-11-06 \| Place change 6: `paint/` branches only move forward for Git clients. The place hook refuses a delete of `refs/heads/paint/*` and any update whose stored tip is not an ancestor of the new tip (`HookObjects::range(stored, [new], 0)`), whatever the client sends (`-o force`, a lease, `--force` through the branch URL); creation and fast-forwards are unchanged, and the server's own writers (merger, expiry, strokes) skip the hook as before. Other client branches keep S2's policy. Replaces "any Git client may delete any such branch" in `demo/place/docs/TILES.md` \| Anyone can push to any branch on the public canvas, so anyone could delete or force-push someone else's `paint/` branch before it merged and drop its tiles; visitors make new branches anyway. Without `-o force`, S2 refuses the delete or rewrite first with its own text (the hook is not asked), which points at `-o force`; with it, the hook's refusal tells the client to push to a new branch (`paint_branches_only_move_forward_for_git_clients`) \|
	1278	+\| 2026-11-06 \| `place` starts with its kill switch released, and the canvas mirror waits for place at start (place change 8, infra half; follow-up to change 7). (1) The environment root no longer has `place_kill_switch` and never sets `PLACE_KILL_SWITCH`; an environment test fails if the place container's environment names it. The variable is removed rather than defaulted to `false` because the live `demo.tfvars` (the maintainers' secret store) sets it to `true`; a stale line there is now only Terraform's undeclared-variable warning. Operators still engage the switch through `POST /api/v1/admin/kill-switch`; since it lives in the task's memory, a restart releases it, and `place_running = false` is the stop that lasts. (2) `canvas-mirror.sh` asks place for `main` (`git ls-remote`) every `MIRROR_START_RETRY` seconds (5) until place names one, for at most one `MIRROR_INTERVAL` of wall time, before its first run, each ask cut off after 10 s (`timeout 10`) so a place that stalls mid-request cannot hold the wait; a `MIRROR_START_RETRY` that is not a positive number of seconds is logged and replaced by 5 (refusing it would stop the mirror, which is not worth a typo); its working repository moves to `${TMPDIR:-/tmp}`. Test: `environment/tests/canvas-mirror.test.sh` (required CI) runs the real script with a place that appears 3 s after start and a 900 s interval: main reaches the GitHub stand-in about 2 s after place answers, where the previous script failed its one fetch at start and waited the interval; a place that never answers ends the wait after one interval and the usual runs go on; `MIRROR_START_RETRY=0` is reported and replaced by 5 (the previous script spun without counting time); a place that takes the request and never answers ends the wait (the previous script hung). Absent `PLACE_KILL_SWITCH` means released: `demo/place/server/src/main.rs` sets `c.kill_switch = var("PLACE_KILL_SWITCH").as_deref() == Some("1")`, where `var` treats unset and empty as `None`, and `PlaceConfig::default` has `kill_switch: false` \| `PLACE_KILL_SWITCH=1` engaged the switch on every deploy and restart, so the public demo was stopped until an operator released it by hand (twice on 2026-11-06), which conflicts with staying open for a public share. The sidecar starts with place (`dependsOn` START) and fetched once before place was listening, so after each restart the mirror lagged a full interval (15 minutes; seen at 16:13 UTC on 2026-11-06). Waiting for `main` rather than for the repository to answer at all also covers a place that answers before it has a `main`. A container health check with `dependsOn` HEALTHY was not taken: place's image (`deploy/docker/Dockerfile`, Debian with only `ca-certificates`) has no HTTP client to probe itself with \|