History

Make tool failures visible telemetry: span outcome attributes, derived error messages, e2e contract (#992)

* Tool failures become visible telemetry: outcome attributes, derived error messages, recording-tracer contracts Expected tool failures (ToolResult.fail) ride the Effect success channel, so the tracer recorded healthy spans while users hit upstream error walls — in a week of prod data, zero of 19.5k plugin.openapi.invoke spans carried http.status_code (annotated inside the inner OpenApi.invoke span, not the wrapper queries target) and 562 of 804 executor.tool.execute error spans had an empty status.message (tagged errors with no message field). - Stamp executor.tool.outcome / error_code / error_status on executor.tool.execute and mcp.tool.dispatch whenever a dispatch resolves to a ToolResult, plus tenant/subject attribution so queries no longer need a trace-id join against the outer request span. - Annotate http.status_code on the plugin.openapi.invoke wrapper span. - Derive messages for the message-less SDK tagged errors and stop formatInvocationCauseMessage rendering plain objects as [object Object]. - Add a recording Tracer to @executor-js/sdk/testing and telemetry contract tests that pin the span names, attributes, and error statuses production queries depend on. * Move product-observability plan note out of the repo * Pin the telemetry contract end-to-end instead of unit-level Replace the recording-tracer unit tests with one e2e scenario that drives the whole production topology — HTTP API → execution engine → sandbox → OpenAPI invoke → a real upstream returning 502 → OTLP export — and asserts the spans as the suite's motel store actually received them: outcome/error_code/error_status + tenant on executor.tool.execute, http.status_code on plugin.openapi.invoke, and ok/fail distinguishable. The unit tests verified span objects in process; the prod bug class this guards against (attribute stamped on a span the exporter never carries, failure riding the success channel) only shows up at the exported layer. Adds a Telemetry e2e service (motel span search with arrival polling), provided when the suite's motel boots (E2E_MOTEL_URL). * Document the telemetry surfaces for future work - prod-telemetry repo skill: the executor-cloud Axiom dataset layout (attributes.custom JSON map, span names and their attributes including the new outcome fields), working APL recipes for error digests and org attribution, prod-DB and PostHog access notes, and the deploy-canary procedure. - e2e/AGENTS.md: the Telemetry service — scenarios can assert on the spans the target actually exported, with the arrival-polling and fixture-tagging gotchas.

Rhys Sullivan committed 5 months ago

Commit 130d0e91 parent 42c0da414 files · +540 −17Browse files

Unified · Split

14 files
  1. .claude/skills/prod-telemetry/SKILL.md +114 −0
  2. e2e/AGENTS.md +25 −0
  3. e2e/cloud/telemetry-contract.test.ts +181 −0
  4. e2e/setup/cloud.globalsetup.ts +4 −0
  5. e2e/src/scenario.ts +7 −1
  6. e2e/src/services.ts +5 −0
  7. e2e/src/surfaces/telemetry.ts +75 −0
  8. packages/core/execution/src/tool-invoker.ts +5 −0
  9. packages/core/sdk/src/elicitation.ts +7 −1
  10. packages/core/sdk/src/errors.ts +52 −9
  11. packages/core/sdk/src/executor.ts +25 −4
  12. packages/core/sdk/src/index.ts +7 −1
  13. packages/core/sdk/src/tool-result.ts +28 −1
  14. packages/plugins/openapi/src/sdk/invoke.ts +5 −0
.claude/skills/prod-telemetry/SKILL.md+114−0
@@ -0,0 +1,114 @@
1---
2name: prod-telemetry
3description: Query Executor's production telemetry — Axiom traces (executor-cloud dataset), prod Postgres via PlanetScale, PostHog product analytics — through the Executor MCP. Use when investigating prod errors, latency, usage, churn signals, or verifying a deploy's telemetry; includes the dataset field layout, working APL recipes, and the error-attribution join.
4---
5
6# Production telemetry access
7
8All three stores are queryable through the Executor MCP's connected
9integrations — no dashboards or credentials needed. Verify the connection
10exists with `connections.list` if a call fails.
11
12## Axiom traces (`axiom_mcp`)
13
14Tool: `axiom_mcp.user.axiomMcpOAuth.querydataset` — the argument is `apl`
15(NOT `query`). Dataset: `['executor-cloud']` (worker spans; browser spans
16join the same traces via traceparent).
17
18**Field layout (the part you'd otherwise rediscover by failed queries):**
19
20- Custom span attributes live under the JSON map `['attributes.custom']`,
21 NOT as top-level `attributes.*` columns. Read with
22 `['attributes.custom']['mcp.tool.name']`. A nonexistent top-level field is
23 a hard query error ("invalid field"), not an empty result.
24- Span status: `['status.code']` (`"OK"`/`"ERROR"`), `['status.message']`.
25- Exceptions: the `events` column carries `exception.type` /
26 `exception.stacktrace` JSON.
27- OTel basics are top-level: `name`, `trace_id`, `span_id`,
28 `parent_span_id`, `duration`, `_time`.
29
30**Span names worth querying** (and their custom attrs):
31
32- `executor.tool.execute` — `mcp.tool.name` (full address), and since
33 PR #992: `executor.tool.outcome` (`ok`/`fail`),
34 `executor.tool.error_code`, `executor.tool.error_status`,
35 `executor.tenant`, `executor.subject`.
36- `mcp.tool.dispatch` — `mcp.tool.name` (sandbox path),
37 `mcp.tool.integration`, same outcome attrs.
38- `plugin.openapi.invoke` — `plugin.openapi.method` / `path_template` /
39 `base_url`, and since PR #992 `http.status_code`.
40- `mcp.request` (outer) — `mcp.auth.organization_id`,
41 `mcp.auth.account_id`, `mcp.tool.name`, CF edge fields (`cf.country`…),
42 MCP client fingerprint (`mcp.client.name`…).
43
44**Recipe — error signatures by class (the daily-digest query):**
45
46```apl
47['executor-cloud']
48| where _time > ago(1d)
49| where ['status.code'] == "ERROR" and name == "executor.tool.execute"
50| extend msg = substring(tostring(['status.message']), 0, 120)
51| extend tool = tostring(['attributes.custom']['mcp.tool.name'])
52| summarize n = count() by msg, tool
53| sort by n desc
54```
55
56**Recipe — attribute errors to orgs.** Tool spans now carry
57`executor.tenant` directly (post-#992). For spans from BEFORE that deploy,
58join through the outer request span:
59
60```apl
61['executor-cloud']
62| where name == "mcp.request" and isnotnull(['attributes.custom']['mcp.auth.organization_id'])
63| project trace_id, org = tostring(['attributes.custom']['mcp.auth.organization_id'])
64| join kind=inner (
65 ['executor-cloud']
66 | where ['status.code'] == "ERROR" and name == "executor.tool.execute"
67 | project trace_id, msg = substring(tostring(['status.message']), 0, 60)
68 ) on trace_id
69| summarize n = count() by org, msg | sort by n desc
70```
71
72**Recipe — upstream failure rate per integration (post-#992 attrs):**
73
74```apl
75['executor-cloud']
76| where _time > ago(1d) and name == "mcp.tool.dispatch"
77| extend outcome = tostring(['attributes.custom']['executor.tool.outcome'])
78| extend integration = tostring(['attributes.custom']['mcp.tool.integration'])
79| where isnotnull(outcome)
80| summarize calls = count(), fails = countif(outcome == "fail") by integration
81| extend failRate = todouble(fails) / todouble(calls)
82| sort by fails desc
83```
84
85**Known signal caveats** (audited 2026-06-12):
86
87- Pre-#992 spans: `ToolResult.fail` outcomes (upstream 4xx/5xx, auth
88 rejections) are INVISIBLE — they rode the Effect success channel with no
89 span marker. Don't conclude "no errors" from old data.
90- Many pre-#992 ERROR spans have an EMPTY `status.message` (tagged errors
91 without a message field) — group those by `events` exception.type instead.
92- `[object Object]` status messages are the pre-#992 formatting bug.
93
94## Prod database (`planetscale_mcp`)
95
96Read tool needs `{organization: "answer-overflow", database: "executor",
97branch: "main"}`. It returns `ok: true` even when the SQL failed — check the
98result text for `Error:`. Use for tenant/integration/connection facts that
99spans don't carry (row sizes, config shapes, counts).
100
101## Product analytics (`posthog_api` / `mcp_posthog_com`)
102
103Browser-side events only (the ~60-event typed catalog, PR #987; server-side
104events not built). The org-key `posthog_api` connection covers the REST API;
105the OAuth MCP connection covers the higher-level tools.
106
107## Verifying a deploy's telemetry (Layer-0 canary)
108
109After deploying telemetry changes: run a known-failing tool call against
110prod, then assert the expected attributes arrive in Axiom within ~1 min.
111Absence of data looks identical to health — query for the NEW attribute
112explicitly rather than eyeballing dashboards. The e2e equivalent runs on
113every suite: `e2e/cloud/telemetry-contract.test.ts` via the `Telemetry`
114service (motel `/api/spans/search?attr.<key>=<value>`).
e2e/AGENTS.md+25−0
@@ -77,6 +77,31 @@ const r = yield * session.call("execute", { code: "return 1 + 1;" }); · 76 unmodified lines
77// human-in-the-loop: session.approvePaused(r.text) resumes a paused execution77// human-in-the-loop: session.approvePaused(r.text) resumes a paused execution
78```78```
79 79
80## Telemetry scenarios (cloud)
81
82The suite boots a motel OTLP store and points the target's real exporter at
83it, so a scenario can assert on the spans the server ACTUALLY exported —
84the layer where "observability silently went dark" bugs live (an attribute
85stamped on a span the exporter never carries looks identical to health).
86
87```ts
88const telemetry = yield * Telemetry; // skips when motel didn't boot
89const span =
90 yield *
91 telemetry.expectSpan({
92 operation: "executor.tool.execute",
93 attributes: { "mcp.tool.name": failAddress }, // exact match, values stringified
94 });
95expect(span.span.tags["executor.tool.outcome"]).toBe("fail");
96```
97
98- `expectSpan` polls (~20s): exporters batch, so arrival is
99 eventually-consistent — "the span reaches the store, soon" IS the contract.
100- Spec gotcha for fixtures: give operations explicit `tags` — tool addresses
101 are `group.leaf`, and an untagged op derives its group from the URL path,
102 so `/fail` does NOT produce a `.fail`-suffixed address.
103- Prior art: `cloud/telemetry-contract.test.ts`.
104
80## Running105## Running
81 106
82```sh107```sh
e2e/cloud/telemetry-contract.test.ts+181−0
@@ -0,0 +1,181 @@
1// Cloud: the telemetry contract, end to end. A tool call that hits an
2// upstream error wall must be visible in the EXPORTED spans — not just
3// handled gracefully for the caller. This is the regression class where the
4// product silently goes dark to operators: `ToolResult.fail` rides the
5// Effect success channel (a healthy-looking span), and an attribute stamped
6// on the wrong span simply never arrives in the trace store, which looks
7// identical to health. So the assertion runs against the OTLP store the dev
8// stack actually exported to (the suite's motel — the same exporter layer
9// that ships prod spans to Axiom), driving the whole production topology:
10// HTTP API → execution engine → sandbox → OpenAPI invoke → a real upstream
11// returning 502 → span batch → OTLP export.
12//
13// Pins two regressions found live in prod (2026-06-12): http.status_code was
14// annotated inside the inner `OpenApi.invoke` span so the `plugin.openapi.
15// invoke` span queries target carried it on 0 of ~19.5k spans; and failed
16// tool calls were indistinguishable from successes on `executor.tool.execute`.
17import { randomBytes } from "node:crypto";
18import { createServer } from "node:http";
19
20import { expect } from "@effect/vitest";
21import { Effect } from "effect";
22import { composePluginApi } from "@executor-js/api/server";
23import { openApiHttpPlugin } from "@executor-js/plugin-openapi/api";
24import { AuthTemplateSlug, ConnectionName, IntegrationSlug } from "@executor-js/sdk/shared";
25
26import { scenario } from "../src/scenario";
27import { Api, Target, Telemetry } from "../src/services";
28
29const api = composePluginApi([openApiHttpPlugin()] as const);
30
31/** Two operations: /ok answers 200, /fail answers 502 — the success and
32 * expected-upstream-failure outcome classes the telemetry must separate. */
33const upstreamSpec = (baseUrl: string): string =>
34 JSON.stringify({
35 openapi: "3.0.3",
36 info: { title: "Telemetry Upstream", version: "1.0.0" },
37 servers: [{ url: baseUrl }],
38 paths: {
39 "/ok": {
40 get: {
41 operationId: "ok",
42 summary: "Succeeds",
43 tags: ["probe"],
44 responses: { "200": { description: "" } },
45 },
46 },
47 "/fail": {
48 get: {
49 operationId: "fail",
50 summary: "Always 502",
51 tags: ["probe"],
52 responses: { "200": { description: "" } },
53 },
54 },
55 },
56 });
57
58/** A real upstream on 127.0.0.1: /ok → 200 JSON, anything else → 502 JSON. */
59const serveUpstream = Effect.acquireRelease(
60 Effect.callback<{ readonly baseUrl: string; readonly close: () => void }>((resume) => {
61 const server = createServer((request, response) => {
62 const ok = request.url?.startsWith("/ok") ?? false;
63 response.writeHead(ok ? 200 : 502, { "content-type": "application/json" });
64 response.end(ok ? '{"fine":true}' : '{"error":{"message":"bad gateway"}}');
65 });
66 server.listen(0, "127.0.0.1", () => {
67 const address = server.address();
68 const port = typeof address === "object" && address ? address.port : 0;
69 resume(
70 Effect.succeed({
71 baseUrl: `http://127.0.0.1:${port}`,
72 close: () => {
73 server.close();
74 server.closeAllConnections();
75 },
76 }),
77 );
78 });
79 }),
80 (server) => Effect.sync(server.close),
81);
82
83scenario(
84 "Telemetry · a failing tool call is visible in the exported spans",
85 { timeout: 180_000 },
86 Effect.scoped(
87 Effect.gen(function* () {
88 const target = yield* Target;
89 const { client: apiClient } = yield* Api;
90 const telemetry = yield* Telemetry;
91 const identity = yield* target.newIdentity();
92 const client = yield* apiClient(api, identity);
93
94 const upstream = yield* serveUpstream;
95
96 // Identifier-safe slug: it becomes a property path in the sandbox code.
97 const slug = IntegrationSlug.make(`telscn${randomBytes(4).toString("hex")}`);
98 yield* client.openapi.addSpec({
99 payload: {
100 spec: { kind: "blob", value: upstreamSpec(upstream.baseUrl) },
101 slug,
102 baseUrl: upstream.baseUrl,
103 authenticationTemplate: [
104 {
105 slug: "apiKey",
106 type: "apiKey",
107 headers: { Authorization: ["Bearer ", { type: "variable", name: "token" }] },
108 },
109 ],
110 },
111 });
112 yield* client.connections.create({
113 payload: {
114 owner: "org",
115 name: ConnectionName.make("main"),
116 integration: slug,
117 template: AuthTemplateSlug.make("apiKey"),
118 value: "telemetry-scenario-token",
119 },
120 });
121
122 const tools = yield* client.tools.list({ query: {} });
123 const addressOf = (op: string) => {
124 const tool = tools.find(
125 (entry) =>
126 String(entry.integration) === String(slug) && String(entry.address).endsWith(`.${op}`),
127 );
128 expect(tool, `the ${op} tool is in the catalog`).toBeDefined();
129 return String(tool!.address);
130 };
131 const failAddress = addressOf("fail");
132 const okAddress = addressOf("ok");
133
134 // Drive both outcome classes through the full production path. The
135 // failing call still completes for the caller — that is exactly why
136 // the exported span is the only place an operator can see it.
137 for (const address of [okAddress, failAddress]) {
138 const execution = yield* client.executions.execute({
139 payload: { code: `return await ${address}({});` },
140 });
141 expect(execution.status, `the ${address} execution completes`).toBe("completed");
142 }
143
144 // The failure: outcome attributes on the tool span...
145 const failSpan = yield* telemetry.expectSpan({
146 operation: "executor.tool.execute",
147 attributes: { "mcp.tool.name": failAddress },
148 });
149 expect(failSpan.span.tags, "a failed tool call is marked on the exported span").toMatchObject(
150 {
151 "executor.tool.outcome": "fail",
152 "executor.tool.error_code": "upstream_http_error",
153 "executor.tool.error_status": "502",
154 },
155 );
156 expect(
157 failSpan.span.tags["executor.tenant"],
158 "the span carries tenant attribution (no trace-join needed to ask 'whose error?')",
159 ).toBeTruthy();
160
161 // ...and the upstream status on the HTTP span queries actually target.
162 const invokeSpan = yield* telemetry.expectSpan({
163 operation: "plugin.openapi.invoke",
164 attributes: { "plugin.openapi.base_url": upstream.baseUrl, "http.status_code": "502" },
165 });
166 expect(
167 invokeSpan.span.tags["plugin.openapi.method"],
168 "the invoke span names the method",
169 ).toBe("GET");
170
171 // The success is distinguishable from the failure.
172 const okSpan = yield* telemetry.expectSpan({
173 operation: "executor.tool.execute",
174 attributes: { "mcp.tool.name": okAddress },
175 });
176 expect(okSpan.span.tags["executor.tool.outcome"], "a successful call is marked ok").toBe(
177 "ok",
178 );
179 }),
180 ),
181);
e2e/setup/cloud.globalsetup.ts+4−0
@@ -28,6 +28,10 @@ export default async function setup(): Promise<(() => Promise<void>) | void> { · 27 unmodified lines
28 28
29 // Suite-owned trace store — every run captures distributed traces.29 // Suite-owned trace store — every run captures distributed traces.
30 const motel = await bootMotel();30 const motel = await bootMotel();
31 // Publish to the test workers (they inherit this process's env): scenarios
32 // that assert on exported spans yield the Telemetry service, which exists
33 // only when this is set. No motel → those scenarios skip, never fail.
34 if (motel) process.env.E2E_MOTEL_URL = motel.url;
31 35
32 const publicUrl = `http://127.0.0.1:${ports.E2E_CLOUD_PORT!}`;36 const publicUrl = `http://127.0.0.1:${ports.E2E_CLOUD_PORT!}`;
33 let booted;37 let booted;
e2e/src/scenario.ts+7−1
@@ -24,6 +24,7 @@ import { makeApiSurface } from "./surfaces/api"; · 23 unmodified lines
24import { makeBrowserSurface } from "./surfaces/browser";24import { makeBrowserSurface } from "./surfaces/browser";
25import { makeCliSurface } from "./surfaces/cli";25import { makeCliSurface } from "./surfaces/cli";
26import { makeMcpSurface } from "./surfaces/mcp";26import { makeMcpSurface } from "./surfaces/mcp";
27import { makeTelemetrySurface } from "./surfaces/telemetry";
27import { completeOAuthConsent, hasOpenCode, makeOpenCodeHome, warmUp } from "./clients/opencode";28import { completeOAuthConsent, hasOpenCode, makeOpenCodeHome, warmUp } from "./clients/opencode";
28import {29import {
29 Api,30 Api,
@@ -35,6 +36,7 @@ import { · 5 unmodified lines
35 Restart,36 Restart,
36 RunDir,37 RunDir,
37 Target,38 Target,
39 Telemetry,
38 TtlControl,40 TtlControl,
39} from "./services";41} from "./services";
40import { buildManifest } from "./viewer/manifest";42import { buildManifest } from "./viewer/manifest";
@@ -62,7 +64,8 @@ type AllServices = · 21 unmodified lines
62 | Billing64 | Billing
63 | OpenCode65 | OpenCode
64 | TtlControl66 | TtlControl
65 | Restart;67 | Restart
68 | Telemetry;
66 69
67/**70/**
68 * What this target on this host can provide. Services beyond the base are71 * What this target on this host can provide. Services beyond the base are
@@ -94,6 +97,9 @@ const contextFor = (target: TargetShape, dir: string): Context.Context<AllServic · 25 unmodified lines
94 if (target.restart) {97 if (target.restart) {
95 context = Context.add(context, Restart, target.restart);98 context = Context.add(context, Restart, target.restart);
96 }99 }
100 if (process.env.E2E_MOTEL_URL) {
101 context = Context.add(context, Telemetry, makeTelemetrySurface(process.env.E2E_MOTEL_URL));
102 }
97 return context;103 return context;
98};104};
99 105
e2e/src/services.ts+5−0
@@ -11,6 +11,7 @@ import type { ApiSurface } from "./surfaces/api"; · 10 unmodified lines
11import type { BrowserSurface } from "./surfaces/browser";11import type { BrowserSurface } from "./surfaces/browser";
12import type { CliSurface } from "./surfaces/cli";12import type { CliSurface } from "./surfaces/cli";
13import type { McpSurface } from "./surfaces/mcp";13import type { McpSurface } from "./surfaces/mcp";
14import type { TelemetrySurface } from "./surfaces/telemetry";
14import type { completeOAuthConsent, makeOpenCodeHome, warmUp } from "./clients/opencode";15import type { completeOAuthConsent, makeOpenCodeHome, warmUp } from "./clients/opencode";
15 16
16/** The target under test (always provided). */17/** The target under test (always provided). */
@@ -34,6 +35,10 @@ export class Mcp extends Context.Service<Mcp, McpSurface>()("e2e/mcp-oauth") {} · 17 unmodified lines
34/** Marker: billing limits are enforced on this target. */35/** Marker: billing limits are enforced on this target. */
35export class Billing extends Context.Service<Billing, true>()("e2e/billing") {}36export class Billing extends Context.Service<Billing, true>()("e2e/billing") {}
36 37
38/** Query the suite's OTLP trace store for spans the target actually exported
39 * (present when the suite booted motel — E2E_MOTEL_URL). */
40export class Telemetry extends Context.Service<Telemetry, TelemetrySurface>()("e2e/telemetry") {}
41
37/** The real OpenCode binary, hermetically driveable (present when installed on this host). */42/** The real OpenCode binary, hermetically driveable (present when installed on this host). */
38export interface OpenCodeClient {43export interface OpenCodeClient {
39 readonly makeHome: typeof makeOpenCodeHome;44 readonly makeHome: typeof makeOpenCodeHome;
e2e/src/surfaces/telemetry.ts+75−0
@@ -0,0 +1,75 @@
1// Telemetry surface: query the suite's motel OTLP store for the spans the
2// target ACTUALLY exported. This is how a scenario asserts the observability
3// contract end-to-end — not "did the code create a span object" but "did the
4// span leave the server, with the attributes production queries depend on".
5// The dangerous regression mode here is silence (an attribute stamped on the
6// wrong span, an error riding the success channel): absent data looks exactly
7// like health, so the contract has to be pinned where the data is read.
8import { Effect, Schedule } from "effect";
9
10/** One exported span, as motel's /api/spans/search returns it. `tags` is the
11 * span's attributes with every value stringified. */
12export interface ExportedSpan {
13 readonly traceId: string;
14 readonly rootOperationName: string;
15 readonly span: {
16 readonly spanId: string;
17 readonly operationName: string;
18 readonly status: "ok" | "error";
19 readonly durationMs: number;
20 readonly tags: Readonly<Record<string, string>>;
21 };
22}
23
24export interface SpanQuery {
25 /** Exact-match attribute filters (motel `attr.<key>=<value>`). */
26 readonly attributes?: Readonly<Record<string, string>>;
27 readonly operation?: string;
28 readonly traceId?: string;
29}
30
31export interface TelemetrySurface {
32 /** One-shot search against the trace store. */
33 readonly searchSpans: (query: SpanQuery) => Effect.Effect<readonly ExportedSpan[], unknown>;
34 /** Search until at least one span matches. Exporters batch (the app
35 * flushes ~1s after the request), so arrival is eventually-consistent —
36 * polling IS the contract: "the span reaches the store, soon". */
37 readonly expectSpan: (query: SpanQuery) => Effect.Effect<ExportedSpan, unknown>;
38}
39
40export const makeTelemetrySurface = (motelUrl: string): TelemetrySurface => {
41 const searchSpans = (query: SpanQuery) =>
42 Effect.gen(function* () {
43 const params = new URLSearchParams({ lookback: "15m", limit: "100" });
44 if (query.operation) params.set("operation", query.operation);
45 if (query.traceId) params.set("traceId", query.traceId);
46 for (const [key, value] of Object.entries(query.attributes ?? {})) {
47 params.set(`attr.${key}`, value);
48 }
49 const response = yield* Effect.promise(() => fetch(`${motelUrl}/api/spans/search?${params}`));
50 if (!response.ok) {
51 return yield* Effect.fail(
52 `motel span search responded ${response.status}: ${yield* Effect.promise(() => response.text())}`,
53 );
54 }
55 const body = (yield* Effect.promise(() => response.json())) as {
56 readonly data?: readonly ExportedSpan[];
57 };
58 return body.data ?? [];
59 });
60
61 return {
62 searchSpans,
63 expectSpan: (query) =>
64 searchSpans(query).pipe(
65 Effect.filterOrFail(
66 (spans) => spans.length > 0,
67 () => `no exported span matched ${JSON.stringify(query)}`,
68 ),
69 Effect.map((spans) => spans[0]!),
70 // ~20s ceiling (40 × 500ms): BatchSpanProcessor flushes ~1s after the
71 // request and the dev stack drains on waitUntil; slower is a real bug.
72 Effect.retry(Schedule.both(Schedule.spaced("500 millis"), Schedule.recurs(40))),
73 ),
74 };
75};
packages/core/execution/src/tool-invoker.ts+5−0
@@ -9,6 +9,7 @@ import type { · 8 unmodified lines
9 ToolSchemaView,9 ToolSchemaView,
10} from "@executor-js/sdk/core";10} from "@executor-js/sdk/core";
11import {11import {
12 annotateToolResultOutcome,
12 authToolFailure,13 authToolFailure,
13 isToolResult,14 isToolResult,
14 ToolResult,15 ToolResult,
@@ -282,6 +283,10 @@ export const makeExecutorToolInvoker = ( · 267 unmodified lines
282 // raw success value and wrapped — keeps the sandbox-facing contract283 // raw success value and wrapped — keeps the sandbox-facing contract
283 // uniform without forcing every tiny test plugin to import284 // uniform without forcing every tiny test plugin to import
284 // `ToolResult.ok`.285 // `ToolResult.ok`.
286 // Expected failures resolve through the success channel, so without the
287 // outcome annotation the dispatch span reads as healthy even when the
288 // caller hit an upstream error or auth wall.
289 yield* annotateToolResultOutcome(result);
285 if (isToolResult(result)) {290 if (isToolResult(result)) {
286 return result;291 return result;
287 }292 }
packages/core/sdk/src/elicitation.ts+7−1
@@ -62,4 +62,10 @@ export class ElicitationDeclinedError extends Schema.TaggedErrorClass<Elicitatio · 61 unmodified lines
62 address: ToolAddress,62 address: ToolAddress,
63 action: Schema.Literals(["decline", "cancel"]),63 action: Schema.Literals(["decline", "cancel"]),
64 },64 },
65) {}65) {
66 // Derived message so telemetry (span status, logs) labels the failure
67 // instead of rendering an Error with an empty message.
68 override get message(): string {
69 return `Tool approval ${this.action === "cancel" ? "cancelled" : "declined"}: ${this.address}`;
70 }
71}
packages/core/sdk/src/errors.ts+52−9
@@ -13,13 +13,24 @@ import { ConnectionName, IntegrationSlug, Owner, ProviderKey, ToolAddress } from · 12 unmodified lines
13// Tool lifecycle13// Tool lifecycle
14// ---------------------------------------------------------------------------14// ---------------------------------------------------------------------------
15 15
16/* Tagged errors without an explicit `message` field define a `message` getter:
17 * `Schema.TaggedErrorClass` instances are real Errors with `message: ""`, and
18 * an empty message propagates everywhere errors are rendered — span
19 * status.message in the tracer, Cause.pretty output, log lines — leaving the
20 * failure unlabeled in telemetry. The getter is derived from the schema fields
21 * (not an own property), so encoding/serialization is unaffected. */
22
16export class ToolNotFoundError extends Schema.TaggedErrorClass<ToolNotFoundError>()(23export class ToolNotFoundError extends Schema.TaggedErrorClass<ToolNotFoundError>()(
17 "ToolNotFoundError",24 "ToolNotFoundError",
18 {25 {
19 address: ToolAddress,26 address: ToolAddress,
20 suggestions: Schema.optional(Schema.Array(ToolAddress)),27 suggestions: Schema.optional(Schema.Array(ToolAddress)),
21 },28 },
22) {}29) {
30 override get message(): string {
31 return `Tool not found: ${this.address}`;
32 }
33}
23 34
24export class ToolInvocationError extends Schema.TaggedErrorClass<ToolInvocationError>()(35export class ToolInvocationError extends Schema.TaggedErrorClass<ToolInvocationError>()(
25 "ToolInvocationError",36 "ToolInvocationError",
@@ -38,7 +49,11 @@ export class ToolBlockedError extends Schema.TaggedErrorClass<ToolBlockedError>( · 12 unmodified lines
38 address: ToolAddress,49 address: ToolAddress,
39 pattern: Schema.String,50 pattern: Schema.String,
40 },51 },
41) {}52) {
53 override get message(): string {
54 return `Tool blocked by policy "${this.pattern}": ${this.address}`;
55 }
56}
42 57
43/** Tool row exists but its owning plugin isn't loaded in this executor config. */58/** Tool row exists but its owning plugin isn't loaded in this executor config. */
44export class PluginNotLoadedError extends Schema.TaggedErrorClass<PluginNotLoadedError>()(59export class PluginNotLoadedError extends Schema.TaggedErrorClass<PluginNotLoadedError>()(
@@ -47,13 +62,21 @@ export class PluginNotLoadedError extends Schema.TaggedErrorClass<PluginNotLoade · 2 unmodified lines
47 address: ToolAddress,62 address: ToolAddress,
48 pluginId: Schema.String,63 pluginId: Schema.String,
49 },64 },
50) {}65) {
66 override get message(): string {
67 return `Plugin "${this.pluginId}" is not loaded for tool: ${this.address}`;
68 }
69}
51 70
52/** Tool was found but its owning plugin has no `invokeTool` handler. */71/** Tool was found but its owning plugin has no `invokeTool` handler. */
53export class NoHandlerError extends Schema.TaggedErrorClass<NoHandlerError>()("NoHandlerError", {72export class NoHandlerError extends Schema.TaggedErrorClass<NoHandlerError>()("NoHandlerError", {
54 address: ToolAddress,73 address: ToolAddress,
55 pluginId: Schema.String,74 pluginId: Schema.String,
56}) {}75}) {
76 override get message(): string {
77 return `Plugin "${this.pluginId}" has no invokeTool handler for tool: ${this.address}`;
78 }
79}
57 80
58// ---------------------------------------------------------------------------81// ---------------------------------------------------------------------------
59// Integration / connection lifecycle82// Integration / connection lifecycle
@@ -62,7 +85,11 @@ export class NoHandlerError extends Schema.TaggedErrorClass<NoHandlerError>()("N · 2 unmodified lines
62export class IntegrationNotFoundError extends Schema.TaggedErrorClass<IntegrationNotFoundError>()(85export class IntegrationNotFoundError extends Schema.TaggedErrorClass<IntegrationNotFoundError>()(
63 "IntegrationNotFoundError",86 "IntegrationNotFoundError",
64 { slug: IntegrationSlug },87 { slug: IntegrationSlug },
65) {}88) {
89 override get message(): string {
90 return `Integration not found: ${this.slug}`;
91 }
92}
66 93
67/** An "add integration" operation targeted a slug (namespace) that is already94/** An "add integration" operation targeted a slug (namespace) that is already
68 * registered. The core `integrations.register` primitive upserts by design95 * registered. The core `integrations.register` primitive upserts by design
@@ -73,14 +100,22 @@ export class IntegrationAlreadyExistsError extends Schema.TaggedErrorClass<Integ · 4 unmodified lines
73 "IntegrationAlreadyExistsError",100 "IntegrationAlreadyExistsError",
74 { slug: IntegrationSlug },101 { slug: IntegrationSlug },
75 { httpApiStatus: 409 },102 { httpApiStatus: 409 },
76) {}103) {
104 override get message(): string {
105 return `Integration already exists: ${this.slug}`;
106 }
107}
77 108
78/** `integrations.remove` was called on an integration declared statically by a109/** `integrations.remove` was called on an integration declared statically by a
79 * plugin at startup (`canRemove: false`). */110 * plugin at startup (`canRemove: false`). */
80export class IntegrationRemovalNotAllowedError extends Schema.TaggedErrorClass<IntegrationRemovalNotAllowedError>()(111export class IntegrationRemovalNotAllowedError extends Schema.TaggedErrorClass<IntegrationRemovalNotAllowedError>()(
81 "IntegrationRemovalNotAllowedError",112 "IntegrationRemovalNotAllowedError",
82 { slug: IntegrationSlug },113 { slug: IntegrationSlug },
83) {}114) {
115 override get message(): string {
116 return `Integration cannot be removed (declared statically by a plugin): ${this.slug}`;
117 }
118}
84 119
85export class ConnectionNotFoundError extends Schema.TaggedErrorClass<ConnectionNotFoundError>()(120export class ConnectionNotFoundError extends Schema.TaggedErrorClass<ConnectionNotFoundError>()(
86 "ConnectionNotFoundError",121 "ConnectionNotFoundError",
@@ -89,7 +124,11 @@ export class ConnectionNotFoundError extends Schema.TaggedErrorClass<ConnectionN · 2 unmodified lines
89 integration: IntegrationSlug,124 integration: IntegrationSlug,
90 name: ConnectionName,125 name: ConnectionName,
91 },126 },
92) {}127) {
128 override get message(): string {
129 return `Connection not found: ${this.integration}.${this.owner}.${this.name}`;
130 }
131}
93 132
94/** A connection create request was rejected before anything was written: the133/** A connection create request was rejected before anything was written: the
95 * input is structurally invalid (no credential inputs for a credentialed134 * input is structurally invalid (no credential inputs for a credentialed
@@ -106,7 +145,11 @@ export class InvalidConnectionInputError extends Schema.TaggedErrorClass<Invalid · 10 unmodified lines
106export class CredentialProviderNotRegisteredError extends Schema.TaggedErrorClass<CredentialProviderNotRegisteredError>()(145export class CredentialProviderNotRegisteredError extends Schema.TaggedErrorClass<CredentialProviderNotRegisteredError>()(
107 "CredentialProviderNotRegisteredError",146 "CredentialProviderNotRegisteredError",
108 { provider: ProviderKey },147 { provider: ProviderKey },
109) {}148) {
149 override get message(): string {
150 return `Credential provider not registered: ${this.provider}`;
151 }
152}
110 153
111/** A connection's value could not be resolved — the provider returned nothing,154/** A connection's value could not be resolved — the provider returned nothing,
112 * or an OAuth token refresh failed and the user must re-auth. */155 * or an OAuth token refresh failed and the user must re-auth. */
packages/core/sdk/src/executor.ts+25−4
@@ -1,4 +1,4 @@
1import { Effect, Layer, Option, Predicate, Schema } from "effect";1import { Effect, Inspectable, Layer, Option, Predicate, Schema } from "effect";
2import { FetchHttpClient, type HttpClient } from "effect/unstable/http";2import { FetchHttpClient, type HttpClient } from "effect/unstable/http";
3import { fumadb } from "@executor-js/fumadb";3import { fumadb } from "@executor-js/fumadb";
4import { memoryAdapter } from "@executor-js/fumadb/adapters/memory";4import { memoryAdapter } from "@executor-js/fumadb/adapters/memory";
@@ -139,6 +139,7 @@ import { · 134 unmodified lines
139 type OAuthEndpointUrlPolicy,139 type OAuthEndpointUrlPolicy,
140} from "./oauth-helpers";140} from "./oauth-helpers";
141import { connectionIdentifier } from "./connection-name-identifier";141import { connectionIdentifier } from "./connection-name-identifier";
142import { annotateToolResultOutcome } from "./tool-result";
142 143
143const PLUGIN_STORAGE_DELETE_KEY_BATCH_SIZE = 90;144const PLUGIN_STORAGE_DELETE_KEY_BATCH_SIZE = 90;
144const MAX_APPROVAL_ARGUMENT_PREVIEW_CHARS = 4_000;145const MAX_APPROVAL_ARGUMENT_PREVIEW_CHARS = 4_000;
@@ -2719,10 +2720,20 @@ export const createExecutor = <const TPlugins extends readonly AnyPlugin[] = rea · 2574 unmodified lines
2719 ): Effect.Effect<unknown, ExecuteError> => {2720 ): Effect.Effect<unknown, ExecuteError> => {
2720 const handler = pickHandler(options);2721 const handler = pickHandler(options);
2721 return Effect.gen(function* () {2722 return Effect.gen(function* () {
2723 // oxlint-disable executor/no-instanceof-error, executor/no-unknown-error-message, executor/no-manual-tag-check -- boundary: normalize arbitrary unknown plugin failures into a human-readable message for ToolInvocationError/telemetry
2722 const formatInvocationCauseMessage = (cause: unknown): string => {2724 const formatInvocationCauseMessage = (cause: unknown): string => {
2723 // oxlint-disable-next-line executor/no-instanceof-error, executor/no-unknown-error-message -- boundary: preserve public execute error message wrapping for unknown plugin failures2725 if (cause instanceof Error && cause.message.length > 0) return cause.message;
2724 return cause instanceof Error ? cause.message : String(cause);2726 // Non-Error / empty-message causes: `String(plainObject)` renders
2727 // "[object Object]", which is what telemetry then shows as the only
2728 // label for the failure. Prefer the tag, else stringify structurally.
2729 if (typeof cause === "object" && cause !== null) {
2730 const tag = (cause as { readonly _tag?: unknown })._tag;
2731 if (typeof tag === "string") return tag;
2732 return Inspectable.toStringUnknown(cause, 0);
2733 }
2734 return String(cause);
2725 };2735 };
2736 // oxlint-enable executor/no-instanceof-error, executor/no-unknown-error-message, executor/no-manual-tag-check
2726 const wrapInvocationError = <A, E>(2737 const wrapInvocationError = <A, E>(
2727 effect: Effect.Effect<A, E>,2738 effect: Effect.Effect<A, E>,
2728 ): Effect.Effect<A, ToolInvocationError> =>2739 ): Effect.Effect<A, ToolInvocationError> =>
@@ -2877,8 +2888,18 @@ export const createExecutor = <const TPlugins extends readonly AnyPlugin[] = rea · 148 unmodified lines
2877 }),2888 }),
2878 );2889 );
2879 }).pipe(2890 }).pipe(
2891 // Expected tool failures (`ToolResult.fail`) resolve through the
2892 // success channel, so the tracer alone would record them as healthy
2893 // spans. Stamp the outcome + error code so telemetry can distinguish
2894 // "tool ran fine" from "user hit an upstream error / auth wall"
2895 // without parsing response bodies.
2896 Effect.tap(annotateToolResultOutcome),
2880 Effect.withSpan("executor.tool.execute", {2897 Effect.withSpan("executor.tool.execute", {
2881 attributes: { "mcp.tool.name": String(address) },2898 attributes: {
2899 "mcp.tool.name": String(address),
2900 "executor.tenant": tenant,
2901 ...(subject != null ? { "executor.subject": subject } : {}),
2902 },
2882 }),2903 }),
2883 );2904 );
2884 };2905 };
packages/core/sdk/src/index.ts+7−1
@@ -324,7 +324,13 @@ export { buildToolTypeScriptPreview } from "./schema-types"; · 323 unmodified lines
324export { InternalError } from "./api-errors";324export { InternalError } from "./api-errors";
325 325
326// ToolResult — typed value-based discriminated union for tool outcomes.326// ToolResult — typed value-based discriminated union for tool outcomes.
327export { ToolResult, isToolResult, type ToolError, type ToolHttpMeta } from "./tool-result";327export {
328 ToolResult,
329 annotateToolResultOutcome,
330 isToolResult,
331 type ToolError,
332 type ToolHttpMeta,
333} from "./tool-result";
328 334
329// Stamped boot-time data-migration ledger for the libSQL-backed apps.335// Stamped boot-time data-migration ledger for the libSQL-backed apps.
330export {336export {
packages/core/sdk/src/tool-result.ts+28−1
@@ -5,7 +5,7 @@ · 4 unmodified lines
5// the Effect failure channel.5// the Effect failure channel.
6// ---------------------------------------------------------------------------6// ---------------------------------------------------------------------------
7 7
8import { Schema } from "effect";8import { Effect, Schema } from "effect";
9 9
10export const ToolErrorSchema = Schema.Struct({10export const ToolErrorSchema = Schema.Struct({
11 code: Schema.String,11 code: Schema.String,
@@ -56,3 +56,30 @@ const isUnknownToolResult = Schema.is(ToolResultSchema); · 44 unmodified lines
56 56
57export const isToolResult = (value: unknown): value is ToolResult<unknown> =>57export const isToolResult = (value: unknown): value is ToolResult<unknown> =>
58 isUnknownToolResult(value);58 isUnknownToolResult(value);
59
60/**
61 * Annotate the current span with the outcome of a tool invocation.
62 *
63 * `ToolResult.fail` rides the Effect *success* channel by design (expected
64 * failures are values, not defects), which means the tracer records those
65 * spans as healthy. Without this, "user keeps hitting 4xx walls" is invisible
66 * to telemetry — the exact class of signal that lets us catch product issues
67 * before they're reported. Stamped attributes:
68 *
69 * - `executor.tool.outcome` — "ok" | "fail" (always, on ToolResults)
70 * - `executor.tool.error_code` — ToolError.code (fail only)
71 * - `executor.tool.error_status` — upstream HTTP status (fail, when present)
72 *
73 * Codes/statuses are enumerable identifiers, never user content — safe span
74 * attributes. Non-ToolResult values (raw success payloads) annotate "ok".
75 */
76export const annotateToolResultOutcome = (value: unknown): Effect.Effect<void> => {
77 if (isToolResult(value) && !value.ok) {
78 return Effect.annotateCurrentSpan({
79 "executor.tool.outcome": "fail",
80 "executor.tool.error_code": value.error.code,
81 ...(value.error.status != null ? { "executor.tool.error_status": value.error.status } : {}),
82 });
83 }
84 return Effect.annotateCurrentSpan({ "executor.tool.outcome": "ok" });
85};
packages/plugins/openapi/src/sdk/invoke.ts+5−0
@@ -625,6 +625,11 @@ export const invokeWithLayer = ( · 624 unmodified lines
625 625
626 return invoke(operation, args, resolvedHeaders, sourceQueryParams).pipe(626 return invoke(operation, args, resolvedHeaders, sourceQueryParams).pipe(
627 Effect.provide(clientWithBaseUrl),627 Effect.provide(clientWithBaseUrl),
628 // `invoke` annotates http.status_code on ITS span (`OpenApi.invoke`,
629 // via Effect.fn) — annotateCurrentSpan inside it never reaches this
630 // wrapper span. Stamp the status here too so queries against
631 // `plugin.openapi.invoke` see the upstream outcome directly.
632 Effect.tap((result) => Effect.annotateCurrentSpan({ "http.status_code": result.status })),
628 Effect.withSpan("plugin.openapi.invoke", {633 Effect.withSpan("plugin.openapi.invoke", {
629 attributes: {634 attributes: {
630 "plugin.openapi.method": operation.method.toUpperCase(),635 "plugin.openapi.method": operation.method.toUpperCase(),

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t