Document the sync lifecycle spans and add a changeset

Rhys Sullivan committed 3 months ago

Commit e8fa5aa1 parent 7ec156e2 files · +38 −8Browse files

Unified · Split

2 files

.changeset/tool-sync-lifecycle.md +13 −0
.claude/skills/prod-telemetry/SKILL.md +25 −8

@@ -0,0 +1,13 @@
	1	+---
	2	+"@executor-js/sdk": patch
	3	+"@executor-js/plugin-mcp": patch
	4	+"@executor-js/plugin-openapi": patch
	5	+---
	6	+
	7	+Fix: a broken integration is no longer re-dialed on every freshness window, and concurrent reads no longer duplicate the same refresh
	8	+
	9	+A tools read refreshes stale tool catalogs before it answers, and the executor stamped the connection's last-synced time even when the listing failed. A server that had been unreachable for a month therefore reported as "synced 30 seconds ago", earned a fresh handshake every freshness window, failed again, and re-stamped — indefinitely. Handshakes that could only ever be refused, because the credential itself had been revoked, were the single largest share of that traffic. Separately, two reads arriving at once had no way to see each other, so both dialed the same server for the same catalog.
	10	+
	11	+Connections now carry a real sync lifecycle. The last-synced stamp is written only by a listing that actually succeeded, so freshness is honest; a drift signal is recorded alongside it rather than erasing it, so "the catalog changed" and "this has never synced" are finally different states with different diagnoses. Failed listings walk a jittered retry ladder that doubles up to a six-hour ceiling, and a listing refused on authentication parks the connection until something that could plausibly fix it happens: an explicit refresh, a connection edit, an OAuth reconnect, or a healthy check. Before dialing, a refresh claims the connection with a leased write token, so exactly one of any number of concurrent readers does the work and the rest answer from the existing catalog; a claimant that loses its lease discards its result instead of overwriting a newer one.
	12	+
	13	+Plugins can now classify an incomplete listing as `auth`, `unreachable`, `protocol`, or `config` via `ResolveToolsResult.incompleteKind`. The MCP plugin reports the first three from the handshake (including reauthorization demands and 401/403 responses, which it previously discarded), and the OpenAPI plugin reports `config`. The tools read's spans gained the claim outcome and per-reason skip counts.

.claude/skills/prod-telemetry/SKILL.md+25−8

@@ -41,18 +41,35 @@ join the same traces via traceparent). · 40 unmodified lines
41	41	`.connection` (present only when the read was filtered),
42	42	`executor.tools.result_count`, and the catalog-refresh counters
43	43	`executor.tools.sync.candidates` (rows the stale scan returned) /
44		− `executor.tools.sync.synced` / `executor.tools.sync.failed`. A slow tools
45		− read is almost always `candidates` > 0: subtract the child span durations to
46		− confirm. A connection stuck permanently stale shows up as `failed` > 0 on
47		− every read for its scope — the refresh is best-effort and never fails the
48		− read, so this counter and the `executor tool catalog refresh failed` warning
49		− are the only signals it emits.
	44	+ `executor.tools.sync.synced` / `executor.tools.sync.failed` /
	45	+ `executor.tools.sync.skipped_claimed` / `.skipped_backoff` /
	46	+ `.skipped_parked`. A slow tools read is almost always `candidates` > 0:
	47	+ subtract the child span durations to confirm. A connection stuck permanently
	48	+ stale shows up as `failed` > 0 on every read for its scope — the refresh is
	49	+ best-effort and never fails the read, so this counter and the
	50	+ `executor tool catalog refresh failed` warning are the only signals it emits.
	51	+
	52	+ The three `skipped_*` counters are how the sync lifecycle reports its work
	53	+ avoidance, and they are the ones to watch after a rollout:
	54	+ `skipped_parked` counts connections whose credential was refused (an `auth`
	55	+ verdict, which no retry can fix), `skipped_backoff` counts ones inside their
	56	+ failure ladder, and `skipped_claimed` counts refreshes another concurrent
	57	+ read had already taken. `candidates` staying high while `synced` collapses
	58	+ toward the skip counters is the intended shape — it means dead upstreams
	59	+ stopped costing handshakes. `skipped_claimed` rising with request concurrency
	60	+ for one integration is cross-isolate deduplication working.
	61	+
50	62	- `executor.tools.sync` (child of the above, one per refreshed connection;
51	63	older spans carry the previous name `executor.tools.sync_stale` and no
52	64	trigger/outcome attrs) — `executor.integration`,
53	65	`executor.connection`, `executor.tools.sync.trigger`
54		− (`stale_marked`/`config_revised`/`expired`) and
55		− `executor.tools.sync.outcome` (`ok`/`fail`). The matching warning log carries
	66	+ (`cold`/`stale_marked`/`config_revised`/`expired`),
	67	+ `executor.tools.sync.claimed` (bool — false means another reader owned the
	68	+ refresh and this one did nothing) and `executor.tools.sync.outcome`
	69	+ (`ok`/`fail`). `cold` split out of `stale_marked` when `tools_stale_at`
	70	+ landed: before that, a never-synced connection and one invalidated
	71	+ mid-invocation were the same row, so spans predating it report every
	72	+ never-synced connection as `stale_marked`. The matching warning log carries
56	73	`integration`, `connection`, `trigger` and `errorTags` only: a refresh runs
57	74	on a live credential, so no cause or upstream message is logged and Axiom
58	75	will not have one to search.