History

Document the sync lifecycle spans and add a changeset

Rhys Sullivan committed 3 months ago

Commit e8fa5aa1 parent 7ec156e2 files · +38 −8Browse files

Unified · Split

2 files
  1. .changeset/tool-sync-lifecycle.md +13 −0
  2. .claude/skills/prod-telemetry/SKILL.md +25 −8
.changeset/tool-sync-lifecycle.md+13−0
@@ -0,0 +1,13 @@
1---
2"@executor-js/sdk": patch
3"@executor-js/plugin-mcp": patch
4"@executor-js/plugin-openapi": patch
5---
6
7**Fix: a broken integration is no longer re-dialed on every freshness window, and concurrent reads no longer duplicate the same refresh**
8
9A tools read refreshes stale tool catalogs before it answers, and the executor stamped the connection's last-synced time even when the listing failed. A server that had been unreachable for a month therefore reported as "synced 30 seconds ago", earned a fresh handshake every freshness window, failed again, and re-stamped — indefinitely. Handshakes that could only ever be refused, because the credential itself had been revoked, were the single largest share of that traffic. Separately, two reads arriving at once had no way to see each other, so both dialed the same server for the same catalog.
10
11Connections now carry a real sync lifecycle. The last-synced stamp is written only by a listing that actually succeeded, so freshness is honest; a drift signal is recorded alongside it rather than erasing it, so "the catalog changed" and "this has never synced" are finally different states with different diagnoses. Failed listings walk a jittered retry ladder that doubles up to a six-hour ceiling, and a listing refused on authentication parks the connection until something that could plausibly fix it happens: an explicit refresh, a connection edit, an OAuth reconnect, or a healthy check. Before dialing, a refresh claims the connection with a leased write token, so exactly one of any number of concurrent readers does the work and the rest answer from the existing catalog; a claimant that loses its lease discards its result instead of overwriting a newer one.
12
13Plugins can now classify an incomplete listing as `auth`, `unreachable`, `protocol`, or `config` via `ResolveToolsResult.incompleteKind`. The MCP plugin reports the first three from the handshake (including reauthorization demands and 401/403 responses, which it previously discarded), and the OpenAPI plugin reports `config`. The tools read's spans gained the claim outcome and per-reason skip counts.
.claude/skills/prod-telemetry/SKILL.md+25−8
@@ -41,18 +41,35 @@ join the same traces via traceparent). · 40 unmodified lines
41 `.connection` (present only when the read was filtered),41 `.connection` (present only when the read was filtered),
42 `executor.tools.result_count`, and the catalog-refresh counters42 `executor.tools.result_count`, and the catalog-refresh counters
43 `executor.tools.sync.candidates` (rows the stale scan returned) /43 `executor.tools.sync.candidates` (rows the stale scan returned) /
44 `executor.tools.sync.synced` / `executor.tools.sync.failed`. A slow tools44 `executor.tools.sync.synced` / `executor.tools.sync.failed` /
45 read is almost always `candidates` > 0: subtract the child span durations to45 `executor.tools.sync.skipped_claimed` / `.skipped_backoff` /
46 confirm. A connection stuck permanently stale shows up as `failed` > 0 on46 `.skipped_parked`. A slow tools read is almost always `candidates` > 0:
47 every read for its scope — the refresh is best-effort and never fails the47 subtract the child span durations to confirm. A connection stuck permanently
48 read, so this counter and the `executor tool catalog refresh failed` warning48 stale shows up as `failed` > 0 on every read for its scope — the refresh is
49 are the only signals it emits.49 best-effort and never fails the read, so this counter and the
50 `executor tool catalog refresh failed` warning are the only signals it emits.
51
52 The three `skipped_*` counters are how the sync lifecycle reports its work
53 avoidance, and they are the ones to watch after a rollout:
54 `skipped_parked` counts connections whose credential was refused (an `auth`
55 verdict, which no retry can fix), `skipped_backoff` counts ones inside their
56 failure ladder, and `skipped_claimed` counts refreshes another concurrent
57 read had already taken. `candidates` staying high while `synced` collapses
58 toward the skip counters is the intended shape — it means dead upstreams
59 stopped costing handshakes. `skipped_claimed` rising with request concurrency
60 for one integration is cross-isolate deduplication working.
61
50- `executor.tools.sync` (child of the above, one per refreshed connection;62- `executor.tools.sync` (child of the above, one per refreshed connection;
51 older spans carry the previous name `executor.tools.sync_stale` and no63 older spans carry the previous name `executor.tools.sync_stale` and no
52 trigger/outcome attrs) — `executor.integration`,64 trigger/outcome attrs) — `executor.integration`,
53 `executor.connection`, `executor.tools.sync.trigger`65 `executor.connection`, `executor.tools.sync.trigger`
54 (`stale_marked`/`config_revised`/`expired`) and66 (`cold`/`stale_marked`/`config_revised`/`expired`),
55 `executor.tools.sync.outcome` (`ok`/`fail`). The matching warning log carries67 `executor.tools.sync.claimed` (bool — false means another reader owned the
68 refresh and this one did nothing) and `executor.tools.sync.outcome`
69 (`ok`/`fail`). `cold` split out of `stale_marked` when `tools_stale_at`
70 landed: before that, a never-synced connection and one invalidated
71 mid-invocation were the same row, so spans predating it report every
72 never-synced connection as `stale_marked`. The matching warning log carries
56 `integration`, `connection`, `trigger` and `errorTags` only: a refresh runs73 `integration`, `connection`, `trigger` and `errorTags` only: a refresh runs
57 on a live credential, so no cause or upstream message is logged and Axiom74 on a live credential, so no cause or upstream message is logged and Axiom
58 will not have one to search.75 will not have one to search.