Encode the drift mark as a token, not a timestamp

Replace connection.tools_stale_at (bigint ms) with tools_stale_token (nullable text). The column answers "was this catalog invalidated since the listing now finishing began" — a version question, which a wall clock answers only as precisely as it ticks. markToolsStale writes a fresh token; an authoritative listing captures the token it observed at its start and clears it by compare-and-set inside replaceCatalog's existing transaction, so a mark that landed mid-listing carries a different token and survives. A missed clear fails closed and converges on the next re-list. This restores the read scan's zero-rows steady state: nothing ever cleared the timestamp column, so the prefilter's drift arm re-selected every once-drifted connection on every read forever. Contract change from a1fdc72d6: the same-millisecond stale/synced tie is gone rather than resolved either way, so the schedule tests that pinned inclusive comparison now pin token presence instead. Deliberate. Also: - produceConnectionTools returns outcome synced/incomplete/lost_claim instead of a boolean, with its own counter and span attribute. A plugin reporting bad news correctly is no longer counted as a sync, nor as a refresh that crashed. writeSyncOutcome reports whether it wrote, so an incomplete listing that had also lost its lease reports lost_claim. - The auth park gates expired only. A never-synced parked connection has no catalog to serve, so it walks the retry ladder instead, which caps the waste and restores recovery for credentials repaired upstream. - markToolsStale runs under the catalog persist lock; it was the one lifecycle write left exposed to enrollment in another fiber's open transaction. Migration 0016 and its snapshot are edited in place: unpushed, never run.

Rhys Sullivan committed 2 months ago

Commit e48fe131 parent a1fdc7211 files · +509 −219Browse files

Unified · Split

11 files

.changeset/tool-sync-lifecycle.md +4 −2
.claude/skills/prod-telemetry/SKILL.md +40 −17
apps/cloud/drizzle/0016_icy_jocasta.sql +1 −1
apps/cloud/drizzle/meta/0016_snapshot.json +3 −3
apps/cloud/src/db/executor-schema.ts +1 −1
packages/core/sdk/src/core-schema.ts +15 −10
packages/core/sdk/src/executor.ts +204 −88
packages/core/sdk/src/plugin.ts +6 −1
packages/core/sdk/src/tool-sync-schedule.test.ts +86 −35
packages/core/sdk/src/tool-sync-schedule.ts +47 −37
packages/core/sdk/src/tools-sync-scope.test.ts +102 −24

.changeset/tool-sync-lifecycle.md+4−2

@@ -8,6 +8,8 @@ · 7 unmodified lines
8	8
9	9	A tools read refreshes stale tool catalogs before it answers, and the executor stamped the connection's last-synced time even when the listing failed. A server that had been unreachable for a month therefore reported as "synced 30 seconds ago", earned a fresh handshake every freshness window, failed again, and re-stamped — indefinitely. Handshakes that could only ever be refused, because the credential itself had been revoked, were the single largest share of that traffic. Separately, two reads arriving at once had no way to see each other, so both dialed the same server for the same catalog.
10	10
11		−Connections now carry a real sync lifecycle. The last-synced stamp is written only by a listing that actually succeeded, so freshness is honest; a drift signal is recorded alongside it rather than erasing it, so "the catalog changed" and "this has never synced" are finally different states with different diagnoses. Failed listings walk a jittered retry ladder that doubles up to a six-hour ceiling, and a listing refused on authentication parks the connection until something that could plausibly fix it happens: an explicit refresh, a connection edit, an OAuth reconnect, a healthy check, an integration config revision, or a fresh drift signal. The park holds off the clock, never an explicit invalidation, so a credential problem caused by integration configuration is retried as soon as that configuration is edited. Before dialing, a refresh claims the connection with a leased write token, so exactly one of any number of concurrent readers does the work and the rest answer from the existing catalog; a claimant that loses its lease discards its result instead of overwriting a newer one.
	11	+Connections now carry a real sync lifecycle. The last-synced stamp is written only by a listing that actually succeeded, so freshness is honest; a drift signal is recorded alongside it rather than erasing it, so "the catalog changed" and "this has never synced" are finally different states with different diagnoses. The drift signal is an opaque token rather than a timestamp, because the question it answers is "was this catalog invalidated since the listing now finishing began" — a version question, which a wall clock can only answer as precisely as it ticks. A listing captures the token it observed at its start and clears it only if that same token is still there, so a signal arriving mid-listing carries a different token, survives, and re-lists on the next read, while a resolved one is genuinely gone and stops being scanned.
12	12
13		−Plugins can now classify an incomplete listing as `auth`, `unreachable`, `protocol`, or `config` via `ResolveToolsResult.incompleteKind`. The MCP plugin reports the first three from the handshake (including reauthorization demands and 401/403 responses, which it previously discarded), and the OpenAPI plugin reports `config`. The tools read's spans gained the claim outcome and per-reason skip counts.
	13	+Failed listings walk a jittered retry ladder that doubles up to a six-hour ceiling. A listing refused on authentication parks the connection, which suppresses only the clock asking a working catalog to re-verify itself: an explicit drift signal, a config revision, and a connection that has never synced at all all still go through, the last because parking a connection with no catalog serves nothing at all until a human notices, and nothing here observes a credential repaired upstream. The park is cleared outright by anything that could plausibly have fixed it — an explicit refresh, a connection edit, an OAuth reconnect, a healthy check, a config revision, or a fresh drift signal. Before dialing, a refresh claims the connection with a leased write token, so exactly one of any number of concurrent readers does the work and the rest answer from the existing catalog; a claimant that loses its lease discards its result instead of overwriting a newer one.
	14	+
	15	+Plugins can now classify an incomplete listing as `auth`, `unreachable`, `protocol`, or `config` via `ResolveToolsResult.incompleteKind`. The MCP plugin reports the first three from the handshake (including reauthorization demands and 401/403 responses, which it previously discarded), and the OpenAPI plugin reports `config`. The tools read's spans report each refresh as one of `synced`, `incomplete`, `lost_claim` or `fail`, with a counter each, so a plugin reporting bad news correctly is no longer indistinguishable from a refresh that crashed or from one that succeeded — alongside the claim outcome and per-reason skip counts.

.claude/skills/prod-telemetry/SKILL.md+40−17

@@ -41,23 +41,42 @@ join the same traces via traceparent). · 40 unmodified lines
41	41	`.connection` (present only when the read was filtered),
42	42	`executor.tools.result_count`, and the catalog-refresh counters
43	43	`executor.tools.sync.candidates` (rows the stale scan returned) /
44		− `executor.tools.sync.synced` / `executor.tools.sync.failed` /
45		− `executor.tools.sync.skipped_claimed` / `.skipped_backoff` /
46		− `.skipped_parked`. A slow tools read is almost always `candidates` > 0:
47		− subtract the child span durations to confirm. A connection stuck permanently
48		− stale shows up as `failed` > 0 on every read for its scope — the refresh is
49		− best-effort and never fails the read, so this counter and the
50		− `executor tool catalog refresh failed` warning are the only signals it emits.
	44	+ `executor.tools.sync.synced` / `.incomplete` / `.failed` / `.lost_claim` /
	45	+ `.skipped_claimed` / `.skipped_backoff` / `.skipped_parked`. A slow tools
	46	+ read is almost always `candidates` > 0: subtract the child span durations to
	47	+ confirm.
	48	+
	49	+ `candidates` is the one to alert on for scan cost: in steady state it is
	50	+ ZERO. Every trigger is cleared by the listing that answers it, drift marks
	51	+ included (`connection.tools_stale_token` is compare-and-set to NULL), so a
	52	+ scope whose `candidates` never returns to 0 has connections that are due and
	53	+ never resolving, not a busy fleet.
	54	+
	55	+ The four terminal counters partition every attempt, and the distinction that
	56	+ matters when triaging is `incomplete` vs `failed`. `incomplete` is the PLUGIN
	57	+ reporting bad news correctly — a refused credential, an unreachable server,
	58	+ an unreadable spec — with the existing catalog kept and the retry ladder
	59	+ advanced. `failed` is the refresh raising or dying before reaching any
	60	+ verdict: the executor's own error channel, swallowed to keep the read
	61	+ answerable, and the thing to page on. A connection stuck permanently stale
	62	+ shows up as one or the other on every read for its scope; `failed` > 0 also
	63	+ emits the `executor tool catalog refresh failed` warning, which is its only
	64	+ other signal. `lost_claim` counts listings that completed and were then
	65	+ discarded because another attempt had re-claimed the connection — wasted
	66	+ upstream work that looks like success from every angle except the write.
51	67
52	68	The three `skipped_*` counters are how the sync lifecycle reports its work
53	69	avoidance, and they are the ones to watch after a rollout:
54	70	`skipped_parked` counts connections whose credential was refused (an `auth`
55		− verdict, which no retry can fix), `skipped_backoff` counts ones inside their
56		− failure ladder, and `skipped_claimed` counts refreshes another concurrent
57		− read had already taken. `candidates` staying high while `synced` collapses
58		− toward the skip counters is the intended shape — it means dead upstreams
59		− stopped costing handshakes. `skipped_claimed` rising with request concurrency
60		− for one integration is cross-isolate deduplication working.
	71	+ verdict) AND whose catalog is merely past its freshness window — the park
	72	+ gates re-verification only, so a never-synced connection with the same
	73	+ verdict reports `skipped_backoff` instead and keeps walking the ladder.
	74	+ `skipped_backoff` counts ones inside their failure ladder, and
	75	+ `skipped_claimed` counts refreshes another concurrent read had already taken.
	76	+ `synced` collapsing toward the skip counters is the intended shape — it means
	77	+ dead upstreams stopped costing handshakes. `skipped_claimed` rising with
	78	+ request concurrency for one integration is cross-isolate deduplication
	79	+ working.
61	80
62	81	- `executor.tools.sync` (child of the above, one per refreshed connection;
63	82	older spans carry the previous name `executor.tools.sync_stale` and no
@@ -66,10 +85,14 @@ join the same traces via traceparent). · 2 unmodified lines
66	85	(`cold`/`stale_marked`/`config_revised`/`expired`),
67	86	`executor.tools.sync.claimed` (bool — false means another reader owned the
68	87	refresh and this one did nothing) and `executor.tools.sync.outcome`
69		− (`ok`/`fail`). `cold` split out of `stale_marked` when `tools_stale_at`
70		− landed: before that, a never-synced connection and one invalidated
71		− mid-invocation were the same row, so spans predating it report every
72		− never-synced connection as `stale_marked`. The matching warning log carries
	88	+ (`synced`/`incomplete`/`lost_claim`/`fail`). Older spans report `ok` where a
	89	+ current one reports `synced` or `incomplete`; that rename is the same
	90	+ distinction as the counters above, so any query matching `outcome == "ok"`
	91	+ needs both spellings to span the change. `cold` split out of `stale_marked`
	92	+ when the drift mark became its own column: before that, a never-synced
	93	+ connection and one invalidated mid-invocation were the same row, so spans
	94	+ predating it report every never-synced connection as `stale_marked`. The
	95	+ matching warning log carries
73	96	`integration`, `connection`, `trigger` and `errorTags` only: a refresh runs
74	97	on a live credential, so no cause or upstream message is logged and Axiom
75	98	will not have one to search.

apps/cloud/drizzle/0016_icy_jocasta.sql+1−1

@@ -1,4 +1,4 @@
1		−ALTER TABLE "connection" ADD COLUMN "tools_stale_at" bigint;--> statement-breakpoint
	1	+ALTER TABLE "connection" ADD COLUMN "tools_stale_token" text;--> statement-breakpoint
2	2	ALTER TABLE "connection" ADD COLUMN "tools_sync_claim_id" text;--> statement-breakpoint
3	3	ALTER TABLE "connection" ADD COLUMN "tools_sync_claim_at" bigint;--> statement-breakpoint
4	4	ALTER TABLE "connection" ADD COLUMN "tools_sync_failures" bigint;--> statement-breakpoint

apps/cloud/drizzle/meta/0016_snapshot.json+3−3

@@ -376,9 +376,9 @@ · 375 unmodified lines
376	376	"primaryKey": false,
377	377	"notNull": false
378	378	},
379		− "tools_stale_at": {
380		− "name": "tools_stale_at",
381		− "type": "bigint",
	379	+ "tools_stale_token": {
	380	+ "name": "tools_stale_token",
	381	+ "type": "text",
382	382	"primaryKey": false,
383	383	"notNull": false
384	384	},

apps/cloud/src/db/executor-schema.ts+1−1

@@ -61,7 +61,7 @@ export const connection = pgTable( · 60 unmodified lines
61	61	description: text("description"),
62	62	last_health: json("last_health"),
63	63	tools_synced_at: bigint("tools_synced_at", { mode: "bigint" }),
64		− tools_stale_at: bigint("tools_stale_at", { mode: "bigint" }),
	64	+ tools_stale_token: text("tools_stale_token"),
65	65	tools_sync_claim_id: text("tools_sync_claim_id"),
66	66	tools_sync_claim_at: bigint("tools_sync_claim_at", { mode: "bigint" }),
67	67	tools_sync_failures: bigint("tools_sync_failures", { mode: "bigint" }),

packages/core/sdk/src/core-schema.ts+15−10

@@ -225,15 +225,20 @@ export const coreTables = defineTables({ · 224 unmodified lines
225	225	// listing: a month-dead MCP server that reads as "synced 30s ago" is
226	226	// re-dialed once per freshness window forever.
227	227	tools_synced_at: nullableBigintColumn("tools_synced_at"),
228		− // Epoch ms an event declared this catalog drifted (an MCP
229		− // `tools/list_changed`, an unknown-tool rejection). Separate from
230		− // `tools_synced_at` so marking a catalog stale no longer destroys the
231		− // last-verified timestamp — "drifted" and "never synced" are different
232		− // states with different diagnoses. OUT-DATED by the next authoritative
233		− // listing rather than cleared by it: the marking fiber and the listing
234		− // fiber are different isolates, so a write that nulled this column would
235		− // erase a drift signal that landed mid-listing.
236		− tools_stale_at: nullableBigintColumn("tools_stale_at"),
	228	+ // Set when an event declares this catalog drifted (an MCP
	229	+ // `tools/list_changed`, an unknown-tool rejection); null when there is no
	230	+ // outstanding drift. Separate from `tools_synced_at` so marking a catalog
	231	+ // stale no longer destroys the last-verified timestamp — "drifted" and
	232	+ // "never synced" are different states with different diagnoses.
	233	+ //
	234	+ // An OPAQUE token, not a timestamp: it answers "was this catalog
	235	+ // invalidated since the listing now finishing began", which is a version
	236	+ // question, and a wall clock answers it only as precisely as it ticks. A
	237	+ // listing captures the token it observed at its start and clears the
	238	+ // column only if that same token is still there, so a mark that landed
	239	+ // mid-listing carries a different token, survives the clear, and re-lists.
	240	+ // Never compared for order.
	241	+ tools_stale_token: nullableTextColumn("tools_stale_token"),
237	242	// The refresh lease. Concurrent reads land in different Workers isolates
238	243	// with no shared memory, so the row IS the coordination medium: a
239	244	// refresh compare-and-sets its own nonce here before dialing, and the
@@ -497,7 +502,7 @@ export const CONNECTION_CATALOG_SCAN_COLUMNS = [ · 257 unmodified lines
497	502	"integration",
498	503	"name",
499	504	"tools_synced_at",
500		− "tools_stale_at",
	505	+ "tools_stale_token",
501	506	"tools_sync_claim_id",
502	507	"tools_sync_claim_at",
503	508	"tools_sync_failures",

packages/core/sdk/src/executor.ts+204−88

@@ -692,7 +692,7 @@ export const DEFAULT_TOOLS_SYNC_TTL_MS = 15 * 60 * 1000; · 691 unmodified lines
692	692
693	693	/** What sent a connection through tool production, for the per-connection span.
694	694	* The read path reports the classified {@link ToolSyncState} verbatim (`cold`
695		− * and `stale_marked` are finally distinguishable, now that `tools_stale_at`
	695	+ * and `stale_marked` are finally distinguishable, now that `tools_stale_token`
696	696	* records a drift without erasing the last-verified stamp); `explicit` is
697	697	* every human-initiated production — create, re-mint, `connections.refresh` —
698	698	* which answers to a caller and so skips the read path's gates entirely. */
@@ -710,33 +710,69 @@ interface ToolSyncAttempt { · 11 unmodified lines
710	710	/** Production asked for by a caller who is waiting for the answer. */
711	711	const EXPLICIT_TOOL_SYNC: ToolSyncAttempt = { trigger: "explicit", claim: null };
712	712
713		−/** What one attempt at producing a connection's tools ended up with.
	713	+/**
	714	+ * What one attempt at producing a connection's tools ended up with. Three
	715	+ * outcomes rather than a boolean, because they are three genuinely different
	716	+ * things to be told and only one of them is a sync:
	717	+ *
	718	+ * - `synced` an AUTHORITATIVE listing landed: the catalog persisted is the
	719	+ * upstream's tool set, and the freshness stamp is honest. A
	720	+ * legitimate empty catalog (no credential bound, or a plugin
	721	+ * with no `resolveTools`) is one of these — "no tools" verified
	722	+ * is still verified.
	723	+ * - `incomplete` the plugin reported it could not produce an authoritative
	724	+ * listing. The existing catalog was KEPT, the failure ladder
	725	+ * advanced, and nothing claimed freshness.
	726	+ * - `lost_claim` the listing completed and then found another attempt had
	727	+ * re-claimed the connection and answered first, so nothing of it
	728	+ * was written at all.
714	729	*
715		− * `lostClaim` is a distinct outcome rather than a flavour of success: the
716		− * listing completed, and then found another attempt had re-claimed the
717		− * connection and answered first, so nothing of it was persisted. Counting that
718		− * as a sync is how a fleet whose refreshes all lose their lease reports as
719		− * fully synced. `tools` is what the caller should answer with either way — the
720		− * persisted catalog when the claim was lost, the fresh listing when it held. */
	730	+ * Collapsing `incomplete` into success is how a fleet of month-dead upstreams
	731	+ * reports as fully synced; collapsing it into `lost_claim` would blame
	732	+ * concurrency for a broken server.
	733	+ *
	734	+ * `tools` is what the caller should answer with in every case — the fresh
	735	+ * listing when it was persisted, the persisted catalog otherwise.
	736	+ */
	737	+type ToolProductionOutcome = "synced" \| "incomplete" \| "lost_claim";
	738	+
721	739	interface ToolProduction {
722	740	readonly tools: readonly Tool[];
723		− readonly lostClaim: boolean;
	741	+ readonly outcome: ToolProductionOutcome;
724	742	}
725	743
726		−/** How much work a read's catalog refresh found and did, for the enclosing
727		− * `executor.tools.list` span. `candidates` is what the stale scan returned —
728		− * the number the scan's filter scoping exists to keep at zero. `failed`
729		− * counts refreshes that were swallowed to keep the read answerable, so a
730		− * connection stuck permanently stale is visible as a rate rather than only
731		− * as an absent `synced`. `lostClaim` counts listings that completed and were
732		− * then discarded because another attempt had re-claimed the connection —
733		− * wasted upstream work, and the one outcome that looks like a success from
734		− * every angle except the write. The three `skipped*` counts are the whole point of
735		− * the lifecycle columns: a fleet whose reads are dominated by `skipped_parked`
736		− * is one where dead credentials stopped costing handshakes. */
	744	+/**
	745	+ * How much work a read's catalog refresh found and did, for the enclosing
	746	+ * `executor.tools.list` span. `candidates` is what the stale scan returned —
	747	+ * the number the scan's filter scoping exists to keep at zero.
	748	+ *
	749	+ * The four terminal counters partition every listing that actually ran — a
	750	+ * candidate that never won its claim is `skippedClaimed`, not a listing — and
	751	+ * each says something the others cannot:
	752	+ *
	753	+ * - `synced` an authoritative listing was persisted (see
	754	+ * {@link ToolProduction}).
	755	+ * - `incomplete` the PLUGIN reported it could not produce a listing — a refused
	756	+ * credential, an unreachable server, an unreadable spec. The
	757	+ * catalog was kept and the ladder advanced.
	758	+ * - `failed` the refresh raised or died before reaching a verdict, and was
	759	+ * swallowed to keep the read answerable. This is the executor's
	760	+ * own error channel, not the upstream's — an `incomplete` is a
	761	+ * connection reporting bad news correctly, a `failed` is one
	762	+ * nobody has a verdict for.
	763	+ * - `lostClaim` the listing completed and was then discarded because another
	764	+ * attempt had re-claimed the connection — wasted upstream work,
	765	+ * and the one outcome that looks like a success from every angle
	766	+ * except the write.
	767	+ *
	768	+ * The three `skipped*` counts are the whole point of the lifecycle columns: a
	769	+ * fleet whose reads are dominated by `skipped_parked` is one where dead
	770	+ * credentials stopped costing handshakes.
	771	+ */
737	772	interface CatalogRefreshSummary {
738	773	readonly candidates: number;
739	774	readonly synced: number;
	775	+ readonly incomplete: number;
740	776	readonly failed: number;
741	777	readonly lostClaim: number;
742	778	readonly skippedClaimed: number;
@@ -747,6 +783,7 @@ interface CatalogRefreshSummary { · 4 unmodified lines
747	783	const NO_CATALOG_REFRESH: CatalogRefreshSummary = {
748	784	candidates: 0,
749	785	synced: 0,
	786	+ incomplete: 0,
750	787	failed: 0,
751	788	lostClaim: 0,
752	789	skippedClaimed: 0,
@@ -773,6 +810,14 @@ const toolSyncJitter = (): number => · 20 unmodified lines
773	810	const toolSyncClaimNonce = (): string =>
774	811	`sync_${Date.now().toString(36)}${Math.random().toString(36).slice(2, 10)}`;
775	812
	813	+/** A drift mark, as an opaque token. Generated exactly like the claim nonce and
	814	+ * for the same reason: it only has to be distinguishable from the value a
	815	+ * concurrent listing observed, which is what makes the clear a compare-and-set
	816	+ * rather than a timestamp comparison that ties inside one millisecond. Never
	817	+ * parsed, never ordered, never an address. */
	818	+const toolsStaleToken = (): string =>
	819	+ `stale_${Date.now().toString(36)}${Math.random().toString(36).slice(2, 10)}`;
	820	+
776	821	/** The error tags in a swallowed refresh cause, as a stable comma-joined set.
777	822	* Tags only: this cause can carry an upstream HTTP request/response, so no
778	823	* message, field or rendered cause from it may reach a log line. A defect
@@ -2669,10 +2714,10 @@ export const createExecutor = <const TPlugins extends readonly AnyPlugin[] = rea · 1890 unmodified lines
2669	2714	// The terminal outcome of a listing that did NOT produce an authoritative
2670	2715	// catalog. `tools_synced_at` is deliberately untouched: stamping it here is
2671	2716	// what made a month-dead server report as "synced 30s ago" and earn a fresh
2672		− // handshake every freshness window. Any drift signal (`tools_stale_at`)
	2717	+ // handshake every freshness window. Any drift signal (`tools_stale_token`)
2673	2718	// also stands, because nothing resolved it — the retry ladder is the only
2674		− // thing holding the next attempt off, and an `auth` verdict parks the
2675		− // connection outright.
	2719	+ // thing holding the next attempt off, and an `auth` verdict parks a
	2720	+ // connection the clock is merely asking to re-verify.
2676	2721	//
2677	2722	// `priorFailures` is the CONSECUTIVE count already on the row, so the
2678	2723	// `failures` derived here is `>= 1` — the contract `scheduleAfterFailure`
@@ -2814,6 +2859,21 @@ export const createExecutor = <const TPlugins extends readonly AnyPlugin[] = rea · 135 unmodified lines
2814	2859	b("name", "=", String(ref.name)),
2815	2860	);
2816	2861
	2862	+ // The row as it stood when this listing began, read before anything is
	2863	+ // dialed. Two things come out of it: the failure count the ladder
	2864	+ // advances from, and the drift token this listing is answering.
	2865	+ const existingRow = yield* findConnectionRow(ref);
	2866	+ const failureSet = (kind: ToolSyncErrorKind \| null, reason: string) =>
	2867	+ toolSyncFailureSet(Number(existingRow?.tools_sync_failures ?? 0), kind, reason);
	2868	+
	2869	+ // THE drift mark this listing observed, or null if there was none. It is
	2870	+ // a version token, not an instant: an authoritative listing clears the
	2871	+ // column only if this exact value is still in it, so a mark written
	2872	+ // while we were dialing carries a different token, survives the clear,
	2873	+ // and re-lists the connection on the next read.
	2874	+ const observedStaleToken =
	2875	+ existingRow?.tools_stale_token == null ? null : String(existingRow.tools_stale_token);
	2876	+
2817	2877	// Does this attempt still own the connection? A read-path refresh takes
2818	2878	// a lease before dialing, and a slow one can lose it: another reader
2819	2879	// sees the lease expire, re-claims, re-lists and finishes first. The
@@ -2854,13 +2914,10 @@ export const createExecutor = <const TPlugins extends readonly AnyPlugin[] = rea · 34 unmodified lines
2854	2914	// report an incomplete listing without classifying it.
2855	2915	//
2856	2916	// The stamp is the instant the listing STARTED, not the instant it
2857		− // landed, and `tools_stale_at` is deliberately NOT cleared. Clearing it
2858		− // was a lost update: `connections.markToolsStale` writes that column
2859		− // from another fiber (a `tools/list_changed` arriving mid-invocation),
2860		− // and a drift signal landing while we were dialing describes a tool set
2861		− // this listing never saw. `classifyToolSync` compares the two stamps
2862		− // instead, so a mark that predates the listing is out-dated by it and
2863		− // one that postdates it survives to re-list.
	2917	+ // landed. It records freshness and NOTHING else: resolving the drift
	2918	+ // mark is a separate compare-and-set inside `replaceCatalog`, so a
	2919	+ // signal that arrived mid-listing is not swallowed by a stamp that
	2920	+ // happens to sort after it.
2864	2921	const successSet = (row: ConnectionRow \| null) => {
2865	2922	const recovering = Number(row?.tools_sync_failures ?? 0) > 0;
2866	2923	return {
@@ -2874,18 +2931,47 @@ export const createExecutor = <const TPlugins extends readonly AnyPlugin[] = rea · 7 unmodified lines
2874	2931	};
2875	2932	};
2876	2933
2877		− // Locked: these run inside a read's concurrent fan-out, and a bare
2878		− // UPDATE issued while another fiber holds an open BEGIN on the shared
2879		− // SQLite connection is enrolled in that fiber's transaction — and lost
2880		− // with it if it rolls back. The permit is NOT reentrant, so nothing
2881		− // reachable from here may take it again.
	2934	+ /** Record a NON-authoritative outcome, or report that the attempt lost
	2935	+ * its lease and wrote nothing. Returns false in the latter case.
	2936	+ *
	2937	+ * Locked: this runs inside a read's concurrent fan-out, and a bare
	2938	+ * UPDATE issued while another fiber holds an open BEGIN on the shared
	2939	+ * SQLite connection is enrolled in that fiber's transaction — and lost
	2940	+ * with it if it rolls back. The permit is NOT reentrant, so nothing
	2941	+ * reachable from here may take it again. Transactional for the same
	2942	+ * reason as {@link replaceCatalog}: `holdsClaim` is only sound read
	2943	+ * inside the transaction that acts on it. */
2882	2944	const writeSyncOutcome = (
2883	2945	set: Record<string, unknown>,
2884		− ): Effect.Effect<void, StorageFailure> =>
2885		− withCatalogPersistLock(core.updateMany("connection", { where: outcomeWhere, set }));
	2946	+ ): Effect.Effect<boolean, StorageFailure> =>
	2947	+ withCatalogPersistLock(
	2948	+ transaction(
	2949	+ Effect.gen(function* () {
	2950	+ if (!(yield* holdsClaim())) return false;
	2951	+ yield* core.updateMany("connection", { where: outcomeWhere, set });
	2952	+ return true;
	2953	+ }),
	2954	+ ),
	2955	+ );
2886	2956
2887		− /** Replace the persisted catalog and record the outcome together, or do
2888		− * neither. Returns false when the attempt lost its lease. */
	2957	+ /** Replace the persisted catalog, record the outcome, and resolve the
	2958	+ * drift mark this listing answered — all three together, or none of
	2959	+ * them. Returns false when the attempt lost its lease.
	2960	+ *
	2961	+ * Already holds the persist permit, which is not reentrant: nothing in
	2962	+ * here may take it again.
	2963	+ *
	2964	+ * INVARIANT on the drift clear: it is a compare-and-set against the
	2965	+ * token observed when this listing STARTED, never an unconditional
	2966	+ * null. `connections.markToolsStale` writes a fresh token from another
	2967	+ * fiber (a `tools/list_changed` arriving mid-invocation), and a mark
	2968	+ * that landed while we were dialing describes a tool set this listing
	2969	+ * never saw — its token differs, the WHERE misses, and the drift
	2970	+ * survives to re-list on the next read.
	2971	+ *
	2972	+ * It fails CLOSED, and it converges: a clear that misses leaves the
	2973	+ * connection drifted, so it re-lists, observes the token that is now
	2974	+ * there, and clears it next time. */
2889	2975	const replaceCatalog = (
2890	2976	toolRows: readonly Record<string, unknown>[],
2891	2977	definitionRows: readonly Record<string, unknown>[],
@@ -2900,18 +2986,31 @@ export const createExecutor = <const TPlugins extends readonly AnyPlugin[] = rea · 8 unmodified lines
2900	2986	yield* core.createMany("tool", toolRows);
2901	2987	yield* core.createMany("definition", definitionRows);
2902	2988	yield* core.updateMany("connection", { where: outcomeWhere, set });
	2989	+ if (observedStaleToken !== null) {
	2990	+ // Keyed on the connection and the token, deliberately NOT on
	2991	+ // `outcomeWhere`: the write above already released the lease
	2992	+ // (`successSet` nulls `tools_sync_claim_id`), so a predicate
	2993	+ // carrying this attempt's claim would match nothing from here
	2994	+ // on. The lease was checked by `holdsClaim` at the top of this
	2995	+ // transaction, which is where it belongs.
	2996	+ yield* core.updateMany("connection", {
	2997	+ where: (b: AnyCb) =>
	2998	+ b.and(connectionWhere(b), b("tools_stale_token", "=", observedStaleToken)),
	2999	+ set: { tools_stale_token: null },
	3000	+ });
	3001	+ }
2903	3002	return true;
2904	3003	}),
2905	3004	),
2906	3005	);
2907	3006
2908	3007	const persistedTools = (
2909		− lostClaim: boolean,
	3008	+ outcome: ToolProductionOutcome,
2910	3009	): Effect.Effect<ToolProduction, StorageFailure> =>
2911	3010	core.findMany("tool", { where }).pipe(
2912	3011	Effect.map((rows) => ({
2913	3012	tools: rows.map((row) => rowToTool(row as ConnectionToolRow)),
2914		− lostClaim,
	3013	+ outcome,
2915	3014	})),
2916	3015	);
2917	3016
@@ -2923,9 +3022,6 @@ export const createExecutor = <const TPlugins extends readonly AnyPlugin[] = rea · 5 unmodified lines
2923	3022	// carry their token outside `item_ids`; no-auth (`"none"` template)
2924	3023	// connections legitimately bind nothing (an empty `item_ids` is their
2925	3024	// canonical shape) — both are exempt.
2926		− const existingRow = yield* findConnectionRow(ref);
2927		− const failureSet = (kind: ToolSyncErrorKind \| null, reason: string) =>
2928		− toolSyncFailureSet(Number(existingRow?.tools_sync_failures ?? 0), kind, reason);
2929	3025	if (
2930	3026	existingRow &&
2931	3027	existingRow.oauth_client == null &&
@@ -2935,17 +3031,17 @@ export const createExecutor = <const TPlugins extends readonly AnyPlugin[] = rea · 3 unmodified lines
2935	3031	// A legitimate clear IS an authoritative listing: the answer is "no
2936	3032	// tools", verified as of now.
2937	3033	if (yield* replaceCatalog([], [], successSet(existingRow))) {
2938		− return { tools: [], lostClaim: false };
	3034	+ return { tools: [], outcome: "synced" };
2939	3035	}
2940		− return yield* persistedTools(true);
	3036	+ return yield* persistedTools("lost_claim");
2941	3037	}
2942	3038
2943	3039	if (!runtime?.plugin.resolveTools) {
2944	3040	// No dynamic tools — clear any existing rows and return empty.
2945	3041	if (yield* replaceCatalog([], [], successSet(existingRow))) {
2946		− return { tools: [], lostClaim: false };
	3042	+ return { tools: [], outcome: "synced" };
2947	3043	}
2948		− return yield* persistedTools(true);
	3044	+ return yield* persistedTools("lost_claim");
2949	3045	}
2950	3046
2951	3047	const result: ResolveToolsResult = yield* runtime.plugin
@@ -2974,14 +3070,18 @@ export const createExecutor = <const TPlugins extends readonly AnyPlugin[] = rea · 22 unmodified lines
2974	3070	// verdict parks the connection outright.
2975	3071	const reason = syncHealthReason(result);
2976	3072	const kind = result.incompleteKind ?? null;
2977		− yield* writeSyncOutcome(failureSet(kind, reason));
	3073	+ // An attempt that had already lost its lease wrote nothing at all —
	3074	+ // not the ladder, not the health detail — so it reports `lost_claim`.
	3075	+ // Reporting `incomplete` there would attribute a verdict to a row the
	3076	+ // winning attempt owns.
	3077	+ const recorded = yield* writeSyncOutcome(failureSet(kind, reason));
2978	3078	yield* Effect.logWarning("executor tool sync preserved catalog", {
2979	3079	reason,
2980	3080	errorKind: kind ?? "unclassified",
2981	3081	integration: String(ref.integration),
2982	3082	connection: String(ref.name),
2983	3083	});
2984		− return yield* persistedTools(false);
	3084	+ return yield* persistedTools(recorded ? "incomplete" : "lost_claim");
2985	3085	}
2986	3086
2987	3087	if (
@@ -2997,16 +3097,20 @@ export const createExecutor = <const TPlugins extends readonly AnyPlugin[] = rea · 9 unmodified lines
2997	3097	// existed is far more often a broken server than a real change, and
2998	3098	// the plugin said nothing about why. Unclassified, so it backs off
2999	3099	// but never parks.
3000		− yield* writeSyncOutcome(failureSet(null, reason));
	3100	+ const recorded = yield* writeSyncOutcome(failureSet(null, reason));
3001	3101	yield* Effect.logWarning("executor tool sync preserved nonzero catalog", {
3002	3102	reason,
3003	3103	integration: String(ref.integration),
3004	3104	connection: String(ref.name),
3005	3105	existingToolCount: keptRows.length,
3006	3106	});
	3107	+ // Re-read on a lost lease: `keptRows` predates the write, and the
	3108	+ // attempt that owns the connection may have replaced the catalog
	3109	+ // since.
	3110	+ if (!recorded) return yield* persistedTools("lost_claim");
3007	3111	return {
3008	3112	tools: keptRows.map((row) => rowToTool(row as ConnectionToolRow)),
3009		− lostClaim: false,
	3113	+ outcome: "incomplete",
3010	3114	};
3011	3115	}
3012	3116	}
@@ -3044,11 +3148,11 @@ export const createExecutor = <const TPlugins extends readonly AnyPlugin[] = rea · 31 unmodified lines
3044	3148	// this catalog with a listing at least as recent as ours. Answer from
3045	3149	// what is persisted rather than overwriting it.
3046	3150	if (!(yield* replaceCatalog(toolRows, definitionRows, successSet(existingRow)))) {
3047		− return yield* persistedTools(true);
	3151	+ return yield* persistedTools("lost_claim");
3048	3152	}
3049	3153
3050	3154	return {
3051		− lostClaim: false,
	3155	+ outcome: "synced",
3052	3156	tools: result.tools.map((tool: ToolDef) =>
3053	3157	rowToTool(
3054	3158	{
@@ -3835,16 +3939,29 @@ export const createExecutor = <const TPlugins extends readonly AnyPlugin[] = rea · 780 unmodified lines
3835	3939	// catalog indistinguishable from one that had never synced. The drift also
3836	3940	// un-parks — the server just told us its tool set moved, which is evidence
3837	3941	// it is reachable and worth one more attempt.
	3942	+ //
	3943	+ // A FRESH token every time, unconditionally overwriting whatever was there.
	3944	+ // That is what makes a mark landing mid-listing survive: the listing clears
	3945	+ // the column only for the token it observed at its start, so overwriting
	3946	+ // with a new one is precisely how this signal outlives the listing that
	3947	+ // could not have seen it.
	3948	+ //
	3949	+ // Locked: this is a lifecycle write like every other, and it can run while
	3950	+ // a read's refresh fan-out holds an open BEGIN on the shared SQLite
	3951	+ // connection — where a bare UPDATE is enrolled in that transaction and lost
	3952	+ // with it if it rolls back. Losing THIS write loses a drift signal outright.
3838	3953	const connectionsMarkToolsStale = (ref: ConnectionRef): Effect.Effect<void, StorageFailure> =>
3839		− core.updateMany("connection", {
3840		− where: (b: AnyCb) =>
3841		− b.and(
3842		− byOwner(ref.owner)(b),
3843		− b("integration", "=", String(ref.integration)),
3844		− b("name", "=", String(ref.name)),
3845		− ),
3846		− set: { tools_stale_at: Date.now(), ...TOOL_SYNC_LADDER_RESET },
3847		− });
	3954	+ withCatalogPersistLock(
	3955	+ core.updateMany("connection", {
	3956	+ where: (b: AnyCb) =>
	3957	+ b.and(
	3958	+ byOwner(ref.owner)(b),
	3959	+ b("integration", "=", String(ref.integration)),
	3960	+ b("name", "=", String(ref.name)),
	3961	+ ),
	3962	+ set: { tools_stale_token: toolsStaleToken(), ...TOOL_SYNC_LADDER_RESET },
	3963	+ }),
	3964	+ );
3848	3965
3849	3966	// ------------------------------------------------------------------
3850	3967	// Active policy source.
@@ -3963,9 +4080,9 @@ export const createExecutor = <const TPlugins extends readonly AnyPlugin[] = rea · 112 unmodified lines
3963	4080	// tool catalog is stale, scoped to that read's own filter — a read for one
3964	4081	// integration never waits on another's upstream handshakes. Four triggers,
3965	4082	// classified by `tool-sync-schedule`:
3966		− // - cold: never synced at all (`tools_synced_at` and `tools_stale_at` both
3967		− // NULL — a fresh row, or one a data migration reset);
3968		− // - stale-marked: `tools_stale_at` postdates the catalog
	4083	+ // - cold: never synced at all (`tools_synced_at` and `tools_stale_token`
	4084	+ // both NULL — a fresh row, or one a data migration reset);
	4085	+ // - stale-marked: `tools_stale_token` is set
3969	4086	// (`connections.markToolsStale` — an MCP `tools/list_changed`
3970	4087	// notification or unknown-tool rejection recorded the drift
3971	4088	// mid-invocation);
@@ -4032,14 +4149,12 @@ export const createExecutor = <const TPlugins extends readonly AnyPlugin[] = rea · 60 unmodified lines
4032	4149	// `blob.getMany`). The two NULL/NOT-NULL arms always stand, so the OR
4033	4150	// can never degenerate to a bare `false`.
4034	4151	//
4035		− // The drift arm is `tools_stale_at IS NOT NULL` rather than a
4036		− // column-to-column comparison against `tools_synced_at`, which the query
4037		− // builder does not offer. A listing OUT-DATES a drift mark rather than
4038		− // clearing it (clearing it lost concurrent marks), so this arm
4039		− // over-selects every connection that has ever drifted and the per-row
4040		− // classifier discards the resolved ones. Deliberate: the alternative is
4041		− // the lost update, and the cost is one projected row per
4042		− // once-drifted connection.
	4152	+ // The drift arm is exactly `tools_stale_token IS NOT NULL`, and it is
	4153	+ // exact rather than an over-selection because an authoritative listing
	4154	+ // CLEARS the token by compare-and-set. That is what keeps the zero-rows
	4155	+ // steady state: encoding the mark as a timestamp nothing ever cleared
	4156	+ // made this arm re-select every once-drifted connection on every read,
	4157	+ // forever.
4043	4158	//
4044	4159	// The revision arm is `<=`, matching `classifyToolSync`: `tools_synced_at`
4045	4160	// is the instant the listing STARTED, so a revision landing in that same
@@ -4054,7 +4169,7 @@ export const createExecutor = <const TPlugins extends readonly AnyPlugin[] = rea · 8 unmodified lines
4054	4169	filter?.connection === undefined ? true : b("name", "=", String(filter.connection)),
4055	4170	b.or(
4056	4171	b.isNull("tools_synced_at"),
4057		− b.isNotNull("tools_stale_at"),
	4172	+ b.isNotNull("tools_stale_token"),
4058	4173	revisedSlugs.length === 0 \|\| latestRevision === null
4059	4174	? false
4060	4175	: b.and(
@@ -4090,7 +4205,8 @@ export const createExecutor = <const TPlugins extends readonly AnyPlugin[] = rea · 29 unmodified lines
4090	4205
4091	4206	const candidate: ToolSyncCandidate = {
4092	4207	toolsSyncedAt: asEpochMs(connection.tools_synced_at),
4093		− toolsStaleAt: asEpochMs(connection.tools_stale_at),
	4208	+ toolsStaleToken:
	4209	+ connection.tools_stale_token == null ? null : String(connection.tools_stale_token),
4094	4210	toolsSyncClaimId:
4095	4211	connection.tools_sync_claim_id == null
4096	4212	? null
@@ -4144,17 +4260,15 @@ export const createExecutor = <const TPlugins extends readonly AnyPlugin[] = rea · 47 unmodified lines
4144	4260	trigger: state,
4145	4261	claim,
4146	4262	}).pipe(
4147		− // `lost_claim` is its own outcome, not a flavour of `ok`: the
4148		− // listing ran, cost an upstream handshake, and was then thrown
4149		− // away because another attempt had re-claimed the connection.
4150		− // Reporting it as synced is how a fleet that persists nothing
4151		− // reads as fully healthy.
4152		− Effect.flatMap((produced) => {
4153		− const outcome = produced.lostClaim ? ("lost_claim" as const) : ("ok" as const);
4154		− return Effect.annotateCurrentSpan({
4155		− "executor.tools.sync.outcome": outcome,
4156		− }).pipe(Effect.as(outcome));
4157		− }),
	4263	+ // Reported verbatim: `synced`, `incomplete` and `lost_claim`
	4264	+ // are three different things to be told, and only the first is
	4265	+ // a sync. Folding the other two into it is how a fleet that
	4266	+ // persists nothing reads as fully healthy.
	4267	+ Effect.flatMap((produced) =>
	4268	+ Effect.annotateCurrentSpan({
	4269	+ "executor.tools.sync.outcome": produced.outcome,
	4270	+ }).pipe(Effect.as(produced.outcome)),
	4271	+ ),
4158	4272	// catchCause, not catch: a plugin DEFECT is not in the error
4159	4273	// channel, and a tools READ must not be failable by one
4160	4274	// connection's misbehaving refresh. The stale-but-working
@@ -4225,7 +4339,8 @@ export const createExecutor = <const TPlugins extends readonly AnyPlugin[] = rea · 64 unmodified lines
4225	4339
4226	4340	return {
4227	4341	candidates: connections.length,
4228		− synced: outcomes.filter((outcome) => outcome === "ok").length,
	4342	+ synced: outcomes.filter((outcome) => outcome === "synced").length,
	4343	+ incomplete: outcomes.filter((outcome) => outcome === "incomplete").length,
4229	4344	failed: outcomes.filter((outcome) => outcome === "fail").length,
4230	4345	lostClaim: outcomes.filter((outcome) => outcome === "lost_claim").length,
4231	4346	skippedClaimed:
@@ -4287,6 +4402,7 @@ export const createExecutor = <const TPlugins extends readonly AnyPlugin[] = rea · 55 unmodified lines
4287	4402	"executor.tools.result_count": tools.length,
4288	4403	"executor.tools.sync.candidates": refresh.candidates,
4289	4404	"executor.tools.sync.synced": refresh.synced,
	4405	+ "executor.tools.sync.incomplete": refresh.incomplete,
4290	4406	"executor.tools.sync.failed": refresh.failed,
4291	4407	"executor.tools.sync.lost_claim": refresh.lostClaim,
4292	4408	"executor.tools.sync.skipped_claimed": refresh.skippedClaimed,

packages/core/sdk/src/plugin.ts+6−1

@@ -237,7 +237,12 @@ export interface PluginCtx<TStore = unknown> { · 236 unmodified lines
237	237	* arrive mid-invocation — e.g. an MCP server sending
238	238	* `notifications/tools/list_changed` or rejecting a call as an unknown
239	239	* tool — where an inline `refresh` would block the caller. The last
240		− * verified sync time is preserved; the drift is recorded alongside it. */
	240	+ * verified sync time is preserved; the drift is recorded alongside it.
	241	+ *
	242	+ * Safe to call at any point, including while a listing for the same
	243	+ * connection is in flight: the mark is recorded as a fresh token, and a
	244	+ * listing only resolves the token it observed when it started. A signal
	245	+ * that arrives mid-listing therefore survives it and re-lists. */
241	246	readonly markToolsStale: (ref: ConnectionRef) => Effect.Effect<void, StorageFailure>;
242	247	/** Resolve a connection's value through its provider (and OAuth refresh).
243	248	* null if the provider can't produce one. */

packages/core/sdk/src/tool-sync-schedule.test.ts+86−35

@@ -28,13 +28,17 @@ import { · 27 unmodified lines
28	28	const NOW = 1_700_000_000_000;
29	29	const TTL = 15 * 60 * 1000;
30	30
	31	+/** A drift mark, as `connections.markToolsStale` writes it. Opaque: nothing
	32	+ * here may read structure out of it, and nothing may order two of them. */
	33	+const STALE_TOKEN = "stale_m3k1x9q2";
	34	+
31	35	/** A row every case overrides into the state it is about. Neutral on purpose:
32	36	* naming it for one of the states under test (it started life as `fresh`)
33	37	* makes `classifyToolSync(fresh({ toolsSyncedAt: null }))` read as a
34	38	* contradiction with the assertion beside it. */
35	39	const candidate = (overrides: Partial<ToolSyncCandidate> = {}): ToolSyncCandidate => ({
36	40	toolsSyncedAt: NOW - 1000,
37		− toolsStaleAt: null,
	41	+ toolsStaleToken: null,
38	42	toolsSyncClaimId: null,
39	43	toolsSyncClaimAt: null,
40	44	toolsSyncFailures: 0,
@@ -72,32 +76,42 @@ describe("classifyToolSync", () => { · 31 unmodified lines
72	76
73	77	it("calls a connection with a drift mark stale_marked, keeping its stamp", () => {
74	78	expect(
75		− classifyToolSync(candidate({ toolsSyncedAt: NOW - 60_000, toolsStaleAt: NOW - 1000 }), NOW),
	79	+ classifyToolSync(
	80	+ candidate({ toolsSyncedAt: NOW - 60_000, toolsStaleToken: STALE_TOKEN }),
	81	+ NOW,
	82	+ ),
76	83	).toBe("stale_marked");
77	84	});
78	85
79		− it("still sees a drift mark that landed in the same millisecond as the stamp", () => {
80		− // Date.now() has millisecond granularity, and a `tools/list_changed`
81		− // arriving inside the millisecond it invalidates must not be swallowed.
82		− expect(classifyToolSync(candidate({ toolsSyncedAt: NOW, toolsStaleAt: NOW }), NOW)).toBe(
83		− "stale_marked",
84		− );
	86	+ it("reads the drift mark as presence, never against the stamp", () => {
	87	+ // The whole point of the token encoding. `tools_stale_token` answers "was
	88	+ // this catalog invalidated since the listing now finishing began", which is
	89	+ // a version question; the earlier encoding asked it of a wall clock, and
	90	+ // `Date.now()` has millisecond granularity, so a mark and a stamp landing in
	91	+ // one millisecond tied. No stamp, however far ahead of the mark, resolves
	92	+ // it — only the compare-and-set that nulls the column does.
	93	+ for (const toolsSyncedAt of [null, 0, NOW - 60_000, NOW, NOW + 60_000]) {
	94	+ expect(
	95	+ classifyToolSync(candidate({ toolsSyncedAt, toolsStaleToken: STALE_TOKEN }), NOW),
	96	+ ).toBe("stale_marked");
	97	+ }
85	98	});
86	99
87		− it("leaves a drift mark the listing out-dated resolved", () => {
88		− // An authoritative listing does not clear `tools_stale_at`, it stamps
89		− // `tools_synced_at` past it. A mark older than the stamp is a settled
90		− // drift, and the connection must not re-list forever because the column is
91		− // still set.
	100	+ it("leaves a connection whose drift mark was cleared resolved", () => {
	101	+ // An authoritative listing clears the token it observed, and a cleared
	102	+ // column is the only thing that settles a drift. This is also what keeps
	103	+ // the read's scan at zero rows in steady state: nothing ever cleared the
	104	+ // old timestamp column, so every connection that had ever drifted was
	105	+ // re-selected on every read, forever.
92	106	expect(
93		− classifyToolSync(candidate({ toolsSyncedAt: NOW - 1000, toolsStaleAt: NOW - 2000 }), NOW),
	107	+ classifyToolSync(candidate({ toolsSyncedAt: NOW - 1000, toolsStaleToken: null }), NOW),
94	108	).toBe("fresh");
95	109	});
96	110
97	111	it("calls a never-synced connection carrying a drift mark stale_marked, not cold", () => {
98		− expect(classifyToolSync(candidate({ toolsSyncedAt: null, toolsStaleAt: NOW }), NOW)).toBe(
99		− "stale_marked",
100		− );
	112	+ expect(
	113	+ classifyToolSync(candidate({ toolsSyncedAt: null, toolsStaleToken: STALE_TOKEN }), NOW),
	114	+ ).toBe("stale_marked");
101	115	});
102	116
103	117	it("calls a catalog older than its integration's config revision config_revised", () => {
@@ -150,7 +164,11 @@ describe("classifyToolSync", () => { · 46 unmodified lines
150	164	it("ranks an explicit drift signal above a config revision and the clock", () => {
151	165	expect(
152	166	classifyToolSync(
153		− candidate({ toolsSyncedAt: NOW - TTL - 1, toolsStaleAt: NOW, configRevisedAt: NOW - 1 }),
	167	+ candidate({
	168	+ toolsSyncedAt: NOW - TTL - 1,
	169	+ toolsStaleToken: STALE_TOKEN,
	170	+ configRevisedAt: NOW - 1,
	171	+ }),
154	172	NOW,
155	173	),
156	174	).toBe("stale_marked");
@@ -215,7 +233,7 @@ describe("isToolSyncBackedOff", () => { · 58 unmodified lines
215	233	const drifted = candidate({
216	234	toolsSyncFailures: 0,
217	235	toolsSyncRetryAt: NOW + TTL,
218		− toolsStaleAt: NOW,
	236	+ toolsStaleToken: STALE_TOKEN,
219	237	});
220	238	expect(isToolSyncBackedOff(drifted, NOW)).toBe(false);
221	239	expect(isSyncEligible(drifted, NOW)).toBe(true);
@@ -243,11 +261,11 @@ describe("decideToolSync", () => { · 21 unmodified lines
243	261	expect(decideToolSync(candidate(), NOW)).toEqual({ kind: "skip", reason: "fresh" });
244	262	});
245	263
246		− it("skips a parked connection however overdue the clock says it is", () => {
	264	+ it("skips a parked connection however far past its freshness window it is", () => {
247	265	expect(
248	266	decideToolSync(
249	267	candidate({
250		− toolsSyncedAt: null,
	268	+ toolsSyncedAt: NOW - 30 * TTL,
251	269	toolsSyncFailures: 9,
252	270	toolsSyncErrorKind: "auth",
253	271	toolsSyncRetryAt: NOW - 1,
@@ -312,7 +330,7 @@ describe("decideToolSync", () => { · 58 unmodified lines
312	330	expect(
313	331	decideToolSync(
314	332	candidate({
315		− toolsSyncedAt: null,
	333	+ toolsSyncedAt: NOW - TTL - 1,
316	334	toolsSyncErrorKind: "auth",
317	335	toolsSyncFailures: 3,
318	336	toolsSyncRetryAt: NOW + TTL,
@@ -322,6 +340,25 @@ describe("decideToolSync", () => { · 3 unmodified lines
322	340	).toEqual({ kind: "skip", reason: "parked" });
323	341	});
324	342
	343	+ it("reports backoff, not parked, for a COLD connection with an auth verdict", () => {
	344	+ // The park does not apply to `cold`, so the ladder is what holds this
	345	+ // connection off — and the reason an operator sees has to say so, because
	346	+ // the read fans the two into different counters and they mean opposite
	347	+ // things: `skipped_parked` is waste permanently avoided, `skipped_backoff`
	348	+ // is a retry that will happen.
	349	+ expect(
	350	+ decideToolSync(
	351	+ candidate({
	352	+ toolsSyncedAt: null,
	353	+ toolsSyncErrorKind: "auth",
	354	+ toolsSyncFailures: 3,
	355	+ toolsSyncRetryAt: NOW + TTL,
	356	+ }),
	357	+ NOW,
	358	+ ),
	359	+ ).toEqual({ kind: "skip", reason: "backoff" });
	360	+ });
	361	+
325	362	it("reports backoff over a live claim", () => {
326	363	expect(
327	364	decideToolSync(
@@ -338,9 +375,10 @@ describe("decideToolSync", () => { · 10 unmodified lines
338	375	});
339	376	});
340	377
341		− // The park answers "the clock alone is not a reason to re-dial a rejected
342		− // credential". It is not an answer to somebody telling us the world changed.
343		− describe("park yields to an explicit invalidation", () => {
	378	+ // The park answers exactly one question: "is re-VERIFYING a catalog we can
	379	+ // still serve worth dialing a credential that was refused". Every other
	380	+ // trigger outranks it.
	381	+ describe("park gates expired and nothing else", () => {
344	382	it("refreshes a parked connection whose integration config was revised", () => {
345	383	// This is the repair path: an `auth` verdict caused by integration
346	384	// configuration is fixed by editing that configuration, and
@@ -361,21 +399,34 @@ describe("decideToolSync", () => { · 14 unmodified lines
361	399	it("refreshes a parked connection that was marked stale", () => {
362	400	expect(
363	401	decideToolSync(
364		− candidate({ toolsSyncedAt: NOW - 60_000, toolsStaleAt: NOW, toolsSyncErrorKind: "auth" }),
	402	+ candidate({
	403	+ toolsSyncedAt: NOW - 60_000,
	404	+ toolsStaleToken: STALE_TOKEN,
	405	+ toolsSyncErrorKind: "auth",
	406	+ }),
365	407	NOW,
366	408	),
367	409	).toEqual({ kind: "refresh", trigger: "stale_marked" });
368	410	});
369	411
370		− it("still parks when only the clock is asking", () => {
371		− for (const overrides of [
372		− { toolsSyncedAt: null },
373		− { toolsSyncedAt: NOW - TTL - 1 },
374		− ] as const) {
375		− expect(
376		− decideToolSync(candidate({ ...overrides, toolsSyncErrorKind: "auth" }), NOW),
377		− ).toEqual({ kind: "skip", reason: "parked" });
378		− }
	412	+ it("refreshes a parked connection that has never synced at all", () => {
	413	+ // A parked `cold` connection has NO catalog to serve, so parking it is
	414	+ // not "keep serving what we have", it is "serve nothing until a human
	415	+ // notices". The ladder still caps the cost at a few dials a day, and it
	416	+ // is what lets a credential repaired upstream — where nothing in this
	417	+ // system observes the repair — recover on its own.
	418	+ expect(
	419	+ decideToolSync(candidate({ toolsSyncedAt: null, toolsSyncErrorKind: "auth" }), NOW),
	420	+ ).toEqual({ kind: "refresh", trigger: "cold" });
	421	+ });
	422	+
	423	+ it("still parks when the clock is only asking for a re-verification", () => {
	424	+ expect(
	425	+ decideToolSync(
	426	+ candidate({ toolsSyncedAt: NOW - TTL - 1, toolsSyncErrorKind: "auth" }),
	427	+ NOW,
	428	+ ),
	429	+ ).toEqual({ kind: "skip", reason: "parked" });
379	430	});
380	431	});
381	432	});

packages/core/sdk/src/tool-sync-schedule.ts+47−37

@@ -46,11 +46,11 @@ export const isToolSyncErrorKind = (value: unknown): value is ToolSyncErrorKind · 45 unmodified lines
46	46	* Every non-`fresh` value is exactly the reason a read has to re-list, so this
47	47	* doubles as the refresh trigger reported on the `executor.tools.sync` span.
48	48	*
49		− * `cold` and `stale_marked` were one value before the `tools_stale_at` column
50		− * existed: `connections.markToolsStale` cleared `tools_synced_at`, which made a
51		− * connection that had never synced indistinguishable from one whose catalog was
52		− * invalidated mid-invocation, and destroyed the last-verified timestamp in the
53		− * process.
	49	+ * `cold` and `stale_marked` were one value before the `tools_stale_token`
	50	+ * column existed: `connections.markToolsStale` cleared `tools_synced_at`, which
	51	+ * made a connection that had never synced indistinguishable from one whose
	52	+ * catalog was invalidated mid-invocation, and destroyed the last-verified
	53	+ * timestamp in the process.
54	54	*/
55	55	export type ToolSyncState = "fresh" \| "cold" \| "stale_marked" \| "config_revised" \| "expired";
56	56
@@ -71,8 +71,11 @@ export interface ToolSyncCandidate { · 14 unmodified lines
71	71	/** Epoch ms of the last AUTHORITATIVE listing. Never stamped by a failed or
72	72	* incomplete one — a month-dead server must not read as "synced 30s ago". */
73	73	readonly toolsSyncedAt: number \| null;
74		− /** Epoch ms an event declared the persisted catalog drifted. */
75		− readonly toolsStaleAt: number \| null;
	74	+ /** The outstanding drift mark, or null when there is none. An opaque token
	75	+ * (see `connection.tools_stale_token`) — its presence is the whole signal
	76	+ * and its value is only ever compared for equality, by the listing that
	77	+ * clears it. */
	78	+ readonly toolsStaleToken: string \| null;
76	79	/** The nonce of the refresh attempt currently holding the write lease. */
77	80	readonly toolsSyncClaimId: string \| null;
78	81	/** Epoch ms the claim was taken, against which the lease expires. */
@@ -118,26 +121,24 @@ export const TOOL_SYNC_JITTER_MAX = 1.2; · 39 unmodified lines
118	121	* Classify a connection's catalog. Total, and ordered by authority: an explicit
119	122	* drift signal outranks a config revision, which outranks the clock.
120	123	*
121		− * BOTH invalidation comparisons are inclusive of the stamp's own millisecond
122		− * (`toolsStaleAt >= syncedAt`, `syncedAt <= configRevisedAt`). `Date.now()` has
123		− * millisecond granularity and `tools_synced_at` records when the listing
124		− * STARTED, so a signal landing in that millisecond describes a change the
125		− * listing cannot have seen. Swallowing it would leave the connection serving a
126		− * catalog that predates the change with no TTL backstop — a plugin without a
127		− * remote catalog never reaches the `expired` branch below.
128		− *
129		− * This is also what resolves a drift mark: an authoritative listing does not
130		− * clear `tools_stale_at`, it out-dates it. Clearing the column was a lost
131		− * update, because the fiber that marked the drift and the fiber finishing the
132		− * listing are different isolates with no coordination between them.
	124	+ * A drift mark is read as PRESENCE, never as an instant: `tools_stale_token` is
	125	+ * an opaque token, and the listing that resolves it clears it by
	126	+ * compare-and-set on the token it observed. Comparing a drift TIMESTAMP against
	127	+ * `tools_synced_at` instead is what made the two stamps landing in one
	128	+ * millisecond ambiguous, and left the column set forever on every connection
	129	+ * that had ever drifted.
	130	+ *
	131	+ * The config-revision comparison stays inclusive of the stamp's own millisecond
	132	+ * (`syncedAt <= configRevisedAt`). `Date.now()` has millisecond granularity and
	133	+ * `tools_synced_at` records when the listing STARTED, so a revision landing in
	134	+ * that millisecond describes a config the listing cannot have read. Swallowing
	135	+ * it would leave the connection serving a catalog that predates the change with
	136	+ * no TTL backstop — a plugin without a remote catalog never reaches the
	137	+ * `expired` branch below.
133	138	*/
134	139	export const classifyToolSync = (candidate: ToolSyncCandidate, now: number): ToolSyncState => {
135	140	const syncedAt = candidate.toolsSyncedAt;
136		− // A never-synced connection sorts before every stamp, which is why the
137		− // comparisons below read through `?? 0` rather than special-casing null.
138		− if (candidate.toolsStaleAt !== null && candidate.toolsStaleAt >= (syncedAt ?? 0)) {
139		− return "stale_marked";
140		− }
	141	+ if (candidate.toolsStaleToken !== null) return "stale_marked";
141	142	if (syncedAt === null) return "cold";
142	143	if (candidate.configRevisedAt !== null && syncedAt <= candidate.configRevisedAt) {
143	144	return "config_revised";
@@ -173,8 +174,8 @@ export const isToolSyncBackedOff = (candidate: ToolSyncCandidate, now: number): · 29 unmodified lines
173	174	/** Whether the connection is parked: its credential is rejected, and no amount
174	175	* of retrying the same grant will change that. Cleared by the human-initiated
175	176	* paths that can actually fix it (explicit refresh, connection update, an
176		− * OAuth re-mint, a healthy probe, a fresh drift signal), and yielded to by
177		− * {@link decideToolSync} whenever an explicit invalidation is outstanding. */
	177	+ * OAuth re-mint, a healthy probe, a fresh drift signal). {@link decideToolSync}
	178	+ * applies it to `expired` alone — every other refresh trigger outranks it. */
178	179	export const isToolSyncParked = (candidate: ToolSyncCandidate): boolean =>
179	180	candidate.toolsSyncErrorKind === "auth";
180	181
@@ -196,15 +197,25 @@ export const isToolSyncParked = (candidate: ToolSyncCandidate): boolean => · 15 unmodified lines
196	197	* decision is the compare-and-set the caller runs against the database, and two
197	198	* readers that both see a free claim here still resolve to one refresh.
198	199	*
199		− * The park yields to an EXPLICIT invalidation. `cold` and `expired` are the
200		− * clock talking, and re-dialing a rejected credential because time passed is
201		− * exactly the waste the park exists to stop; `stale_marked` and
202		− * `config_revised` are somebody telling us the world changed. A config revision
203		− * in particular is how an `auth` verdict caused by integration configuration
204		− * (the key in the wrong place, the wrong token endpoint) gets FIXED, and
205		− * `integrations.update` is the one invalidation path that does not clear the
206		− * ladder, so swallowing it here would leave the repaired connection parked
207		− * forever.
	200	+ * The park gates `expired` and NOTHING else. `expired` is the only trigger that
	201	+ * is purely the clock asking a connection which already has a working catalog
	202	+ * to re-verify it, and re-dialing a rejected credential to re-verify a catalog
	203	+ * we can still serve is exactly the waste the park exists to stop.
	204	+ *
	205	+ * `stale_marked` and `config_revised` are somebody telling us the world
	206	+ * changed. A config revision in particular is how an `auth` verdict caused by
	207	+ * integration configuration (the key in the wrong place, the wrong token
	208	+ * endpoint) gets FIXED, and `integrations.update` is the one invalidation path
	209	+ * that does not clear the ladder, so swallowing it would leave the repaired
	210	+ * connection parked forever.
	211	+ *
	212	+ * `cold` is deliberately NOT parked, and that is a change from parking every
	213	+ * clock-driven trigger. A parked connection that has never synced has no
	214	+ * catalog to serve, so parking it is not "keep serving what we have", it is
	215	+ * "serve nothing, forever, until a human notices". The retry ladder already
	216	+ * caps a never-synced auth failure at roughly four dials a day, and letting the
	217	+ * clock drive it back is what restores automatic recovery when the credential
	218	+ * is repaired upstream — where nothing in this system observes the repair.
208	219	*/
209	220	export type ToolSyncDecision =
210	221	\| { readonly kind: "skip"; readonly reason: ToolSyncSkip }
@@ -213,8 +224,7 @@ export type ToolSyncDecision = · 2 unmodified lines
213	224	export const decideToolSync = (candidate: ToolSyncCandidate, now: number): ToolSyncDecision => {
214	225	const state = classifyToolSync(candidate, now);
215	226	if (state === "fresh") return { kind: "skip", reason: "fresh" };
216		− const timeDriven = state === "cold" \|\| state === "expired";
217		− if (timeDriven && isToolSyncParked(candidate)) return { kind: "skip", reason: "parked" };
	227	+ if (state === "expired" && isToolSyncParked(candidate)) return { kind: "skip", reason: "parked" };
218	228	if (isToolSyncBackedOff(candidate, now)) return { kind: "skip", reason: "backoff" };
219	229	if (isToolSyncClaimLive(candidate, now)) return { kind: "skip", reason: "claimed" };
220	230	return { kind: "refresh", trigger: state };

packages/core/sdk/src/tools-sync-scope.test.ts+102−24

@@ -2,7 +2,7 @@ import { describe, expect, it } from "@effect/vitest"; · 1 unmodified line
2	2	import { Duration, Effect, Fiber, Latch } from "effect";
3	3
4	4	import { CONNECTION_CATALOG_SCAN_COLUMNS } from "./core-schema";
5		−import { createExecutor } from "./executor";
	5	+import { DEFAULT_TOOLS_SYNC_TTL_MS, createExecutor } from "./executor";
6	6	import type { FumaDb } from "./fuma-runtime";
7	7	import { AuthTemplateSlug, ConnectionName, IntegrationSlug, ToolName } from "./ids";
8	8	import { definePlugin } from "./plugin";
@@ -58,6 +58,11 @@ const makeCountingPlugin = () => { · 49 unmodified lines
58	58	const plugin = definePlugin(() => ({
59	59	id: "counting" as const,
60	60	storage: () => ({}),
	61	+ // Stands in for an MCP server: a catalog only the upstream knows, so the
	62	+ // freshness TTL can expire it. Without this the `expired` state is
	63	+ // unreachable from these tests, and `expired` is the one state the park
	64	+ // gates.
	65	+ remoteToolCatalog: true,
61	66	resolveTools: (input) =>
62	67	Effect.gen(function* () {
63	68	const slug = String(input.connection.integration);
@@ -214,7 +219,7 @@ const SCAN_PROJECTION = [ · 150 unmodified lines
214	219	"integration",
215	220	"name",
216	221	"tools_synced_at",
217		− "tools_stale_at",
	222	+ "tools_stale_token",
218	223	"tools_sync_claim_id",
219	224	"tools_sync_claim_at",
220	225	"tools_sync_failures",
@@ -255,7 +260,7 @@ const makeHarness = () => · 34 unmodified lines
255	260	/** Clear every connection's catalog stamp, which is the `cold` trigger:
256	261	* a row with no stamp and no drift mark has never synced. NOT the
257	262	* drift trigger — `connections.markToolsStale` leaves `tools_synced_at`
258		− * alone and writes `tools_stale_at`, which is what
	263	+ * alone and writes `tools_stale_token`, which is what
259	264	* `executor.counting.markStale` drives. Used by the cases that only
260	265	* need a connection to be due, whatever made it due. */
261	266	clearEveryCatalogStamp: () =>
@@ -265,6 +270,17 @@ const makeHarness = () => · 3 unmodified lines
265	270	set: { tools_synced_at: null },
266	271	}),
267	272	),
	273	+ /** Push every catalog past the freshness window, which is the `expired`
	274	+ * trigger: a connection that HAS a working catalog and is only being
	275	+ * asked to re-verify it. Distinct from `cold` in exactly the way the
	276	+ * park cares about, so the two cannot share a helper. */
	277	+ expireEveryCatalog: () =>
	278	+ Effect.promise(() =>
	279	+ observed.db.updateMany("connection", {
	280	+ where: (b) => b.isNotNull("tools_synced_at"),
	281	+ set: { tools_synced_at: Date.now() - DEFAULT_TOOLS_SYNC_TTL_MS - 1 },
	282	+ }),
	283	+ ),
268	284	/** Push every catalog stamp into the past. The `config_revised` trigger
269	285	* compares `tools_synced_at` against a stamp taken later in the same
270	286	* test; on an in-memory database both can land in one millisecond and
@@ -294,7 +310,7 @@ const makeHarness = () => · 23 unmodified lines
294	310	? Effect.die(`no connection row named ${name}`)
295	311	: Effect.succeed({
296	312	syncedAt: row.tools_synced_at == null ? null : Number(row.tools_synced_at),
297		− staleAt: row.tools_stale_at == null ? null : Number(row.tools_stale_at),
	313	+ staleToken: row.tools_stale_token ?? null,
298	314	failures: row.tools_sync_failures == null ? 0 : Number(row.tools_sync_failures),
299	315	retryAt:
300	316	row.tools_sync_retry_at == null ? null : Number(row.tools_sync_retry_at),
@@ -513,7 +529,7 @@ describe("tools read catalog refresh lifecycle", () => { · 212 unmodified lines
513	529	const harness = yield* makeHarness();
514	530	yield* harness.connect(INTEG_A, "main");
515	531	// Genuinely drifted, through the path a plugin actually uses: the
516		− // connection keeps its stamp and carries a newer `tools_stale_at`.
	532	+ // connection keeps its stamp and carries a `tools_stale_token`.
517	533	yield* harness.executor.counting.markStale(INTEG_A, "main");
518	534	harness.renameToolsTo("redeploy");
519	535	harness.resolved.length = 0;
@@ -612,7 +628,10 @@ describe("tools read catalog refresh lifecycle", () => { · 92 unmodified lines
612	628	Effect.gen(function* () {
613	629	const harness = yield* makeHarness();
614	630	yield* harness.connect(INTEG_A, "main");
615		− yield* harness.clearEveryCatalogStamp();
	631	+ // EXPIRED, not cold: the connection has a working catalog and the clock
	632	+ // is only asking it to re-verify. That is the one thing the park gates,
	633	+ // and the case below pins the other side of the rule.
	634	+ yield* harness.expireEveryCatalog();
616	635	harness.resolveIncompleteAs("auth");
617	636	harness.resolved.length = 0;
618	637
@@ -621,8 +640,9 @@ describe("tools read catalog refresh lifecycle", () => { · 2 unmodified lines
621	640	expect((yield* harness.syncStateOf("main")).errorKind).toBe("auth");
622	641
623	642	// Parked: not even elapsing the backoff window lets a read dial again.
624		− // Retrying a rejected credential cannot change the verdict, and 401
625		− // handshakes were the largest single source of wasted upstream calls.
	643	+ // Retrying a rejected credential to re-verify a catalog we can still
	644	+ // serve cannot change the verdict, and 401 handshakes were the largest
	645	+ // single source of wasted upstream calls.
626	646	yield* harness.elapseEveryBackoff();
627	647	yield* harness.executor.tools.list({ integration: INTEG_A });
628	648	expect(harness.resolved).toEqual(["alpha/main"]);
@@ -645,6 +665,45 @@ describe("tools read catalog refresh lifecycle", () => { · 16 unmodified lines
645	665	),
646	666	);
647	667
	668	+ it.effect("walks the ladder for a never-synced connection whose credential was refused", () =>
	669	+ Effect.scoped(
	670	+ Effect.gen(function* () {
	671	+ const harness = yield* makeHarness();
	672	+ yield* harness.connect(INTEG_A, "main");
	673	+ // COLD: no catalog at all. Parking this serves nothing, forever, until
	674	+ // a human notices — and nothing in this system observes a credential
	675	+ // repaired upstream, so the clock has to be what brings it back.
	676	+ yield* harness.clearEveryCatalogStamp();
	677	+ harness.resolveIncompleteAs("auth");
	678	+ harness.resolved.length = 0;
	679	+
	680	+ yield* harness.executor.tools.list({ integration: INTEG_A });
	681	+ expect(harness.resolved).toEqual(["alpha/main"]);
	682	+ expect((yield* harness.syncStateOf("main")).errorKind).toBe("auth");
	683	+
	684	+ // The LADDER holds it off, not the park — so reads inside the window
	685	+ // still cost nothing.
	686	+ yield* harness.executor.tools.list({ integration: INTEG_A });
	687	+ expect(harness.resolved).toEqual(["alpha/main"]);
	688	+
	689	+ // Past the rung, a credential fixed upstream is picked up with no human
	690	+ // in the loop. Under a park that gated `cold` this read never dials and
	691	+ // the connection advertises no tools indefinitely.
	692	+ harness.resolveIncompleteAs(undefined);
	693	+ yield* harness.elapseEveryBackoff();
	694	+
	695	+ const tools = yield* harness.executor.tools.list({ integration: INTEG_A });
	696	+ expect(harness.resolved).toEqual(["alpha/main", "alpha/main"]);
	697	+ expect(tools.map((tool) => String(tool.name))).toEqual(["alpha_deploy"]);
	698	+
	699	+ const recovered = yield* harness.syncStateOf("main");
	700	+ expect(recovered.errorKind).toBeNull();
	701	+ expect(recovered.failures).toBe(0);
	702	+ expect(recovered.syncedAt).not.toBeNull();
	703	+ }),
	704	+ ),
	705	+ );
	706	+
648	707	it.effect("keeps retrying a parked connection whose integration config was revised", () =>
649	708	Effect.scoped(
650	709	Effect.gen(function* () {
@@ -696,7 +755,7 @@ describe("tools read catalog refresh lifecycle", () => { · 45 unmodified lines
696	755
697	756	const synced = yield* harness.syncStateOf("main");
698	757	expect(synced.syncedAt).not.toBeNull();
699		− expect(synced.staleAt).toBeNull();
	758	+ expect(synced.staleToken).toBeNull();
700	759
701	760	yield* harness.executor.counting.markStale(INTEG_A, "main");
702	761
@@ -704,7 +763,7 @@ describe("tools read catalog refresh lifecycle", () => { · 1 unmodified line
704	763	// Both facts survive: when it last verified, and that it has since
705	764	// drifted. Clearing the stamp used to conflate this with "never synced".
706	765	expect(drifted.syncedAt).toBe(synced.syncedAt);
707		− expect(drifted.staleAt).not.toBeNull();
	766	+ expect(drifted.staleToken).not.toBeNull();
708	767
709	768	harness.renameToolsTo("redeploy");
710	769	harness.resolved.length = 0;
@@ -713,20 +772,26 @@ describe("tools read catalog refresh lifecycle", () => { · 2 unmodified lines
713	772	expect(harness.resolved).toEqual(["alpha/main"]);
714	773	expect(tools.map((tool) => String(tool.name))).toEqual(["alpha_redeploy"]);
715	774
716		− // The re-list settled the drift by OUT-DATING the mark, not by clearing
717		− // it: the stamp is now at or past `tools_stale_at`, which is what
718		− // `classifyToolSync` reads. Clearing the column instead is the lost
719		− // update the case below is about.
	775	+ // The re-list settled the drift by CLEARING the token it observed. It
	776	+ // is a compare-and-set, not an unconditional null (the case below is
	777	+ // the mark it must not clear), and the clear is what makes the
	778	+ // resolution observable rather than inferred from two timestamps that
	779	+ // can land in the same millisecond.
720	780	const settled = yield* harness.syncStateOf("main");
721		− expect(settled.staleAt).not.toBeNull();
	781	+ expect(settled.staleToken).toBeNull();
722	782	expect(settled.syncedAt).not.toBeNull();
723		− expect(Number(settled.syncedAt)).toBeGreaterThanOrEqual(Number(settled.staleAt));
724	783
725		− // And the connection is genuinely done: the next read neither scans it
726		− // as due nor dials it again.
	784	+ // And the connection is genuinely done: the next read does not dial
	785	+ // it — and, the property the token encoding exists for, does not even
	786	+ // SCAN it. A drift mark nothing ever clears leaves the scan's
	787	+ // `tools_stale_token IS NOT NULL` arm matching this row on every read
	788	+ // for the rest of its life, which is the steady state PR 1 established
	789	+ // and its `candidates` gauge measures.
727	790	harness.resolved.length = 0;
	791	+ harness.connectionScans.length = 0;
728	792	yield* harness.executor.tools.list({ integration: INTEG_A });
729	793	expect(harness.resolved).toEqual([]);
	794	+ expect(harness.connectionScans).toEqual([{ rows: 0, select: SCAN_PROJECTION }]);
730	795	}),
731	796	),
732	797	);
@@ -737,6 +802,7 @@ describe("tools read catalog refresh lifecycle", () => { · 4 unmodified lines
737	802	const harness = yield* makeHarness();
738	803	yield* harness.connect(INTEG_A, "main");
739	804	yield* harness.executor.counting.markStale(INTEG_A, "main");
	805	+ const observed = yield* harness.syncStateOf("main");
740	806	harness.renameToolsTo("redeploy");
741	807	harness.resolved.length = 0;
742	808	harness.blockResolves();
@@ -751,16 +817,20 @@ describe("tools read catalog refresh lifecycle", () => { · 8 unmodified lines
751	817	// different fiber, no coordination with the one in flight, and it
752	818	// describes a tool set this listing cannot have seen.
753	819	yield* harness.executor.counting.markStale(INTEG_A, "main");
	820	+ const marked = yield* harness.syncStateOf("main");
	821	+ // A FRESH token, which is the entire mechanism: the listing will clear
	822	+ // only the token it observed at its start, and this is not that token.
	823	+ expect(marked.staleToken).not.toBe(observed.staleToken);
754	824
755	825	harness.releaseResolves();
756	826	yield* Fiber.join(listing);
757	827
758		− // The mark survived the success write. Nulling `tools_stale_at` here is
759		− // a lost update, and for a plugin with no remote catalog nothing else
760		− // would ever re-list the connection.
	828	+ // The mark survived the success write, verbatim. Nulling
	829	+ // `tools_stale_token` unconditionally here is a lost update, and for a
	830	+ // plugin with no remote catalog nothing else would ever re-list the
	831	+ // connection.
761	832	const state = yield* harness.syncStateOf("main");
762		− expect(state.staleAt).not.toBeNull();
763		− expect(Number(state.staleAt)).toBeGreaterThanOrEqual(Number(state.syncedAt));
	833	+ expect(state.staleToken).toBe(marked.staleToken);
764	834
765	835	// Which is the fact that matters: the next read dials again.
766	836	harness.renameToolsTo("rerelease");
@@ -768,6 +838,12 @@ describe("tools read catalog refresh lifecycle", () => { · 1 unmodified line
768	838	const tools = yield* harness.executor.tools.list({ integration: INTEG_A });
769	839	expect(harness.resolved).toEqual(["alpha/main"]);
770	840	expect(tools.map((tool) => String(tool.name))).toEqual(["alpha_rerelease"]);
	841	+
	842	+ // And it CONVERGES. That read observed the surviving token and cleared
	843	+ // it, so the drift is settled in one extra listing rather than leaving
	844	+ // the connection permanently marked — a clear that misses costs one
	845	+ // re-list, never a standing scan.
	846	+ expect((yield* harness.syncStateOf("main")).staleToken).toBeNull();
771	847	}),
772	848	),
773	849	);
@@ -799,7 +875,9 @@ describe("tools read catalog refresh lifecycle", () => { · 25 unmodified lines
799	875	expect(state.syncedAt).toBe(synced.syncedAt);
800	876	expect(state.failures).toBe(4);
801	877	expect(state.errorKind).toBe("unreachable");
802		− expect(state.staleAt).not.toBeNull();
	878	+ // The drift mark also stands: only an authoritative listing clears it,
	879	+ // and none of these four were one.
	880	+ expect(state.staleToken).not.toBeNull();
803	881	}),
804	882	),
805	883	);