* Tool failures become visible telemetry: outcome attributes, derived error messages, recording-tracer contracts Expected tool failures (ToolResult.fail) ride the Effect success channel, so the tracer recorded healthy spans while users hit upstream error walls — in a week of prod data, zero of 19.5k plugin.openapi.invoke spans carried http.status_code (annotated inside the inner OpenApi.invoke span, not the wrapper queries target) and 562 of 804 executor.tool.execute error spans had an empty status.message (tagged errors with no message field). - Stamp executor.tool.outcome / error_code / error_status on executor.tool.execute and mcp.tool.dispatch whenever a dispatch resolves to a ToolResult, plus tenant/subject attribution so queries no longer need a trace-id join against the outer request span. - Annotate http.status_code on the plugin.openapi.invoke wrapper span. - Derive messages for the message-less SDK tagged errors and stop formatInvocationCauseMessage rendering plain objects as [object Object]. - Add a recording Tracer to @executor-js/sdk/testing and telemetry contract tests that pin the span names, attributes, and error statuses production queries depend on. * Move product-observability plan note out of the repo * Pin the telemetry contract end-to-end instead of unit-level Replace the recording-tracer unit tests with one e2e scenario that drives the whole production topology — HTTP API → execution engine → sandbox → OpenAPI invoke → a real upstream returning 502 → OTLP export — and asserts the spans as the suite's motel store actually received them: outcome/error_code/error_status + tenant on executor.tool.execute, http.status_code on plugin.openapi.invoke, and ok/fail distinguishable. The unit tests verified span objects in process; the prod bug class this guards against (attribute stamped on a span the exporter never carries, failure riding the success channel) only shows up at the exported layer. Adds a Telemetry e2e service (motel span search with arrival polling), provided when the suite's motel boots (E2E_MOTEL_URL). * Document the telemetry surfaces for future work - prod-telemetry repo skill: the executor-cloud Axiom dataset layout (attributes.custom JSON map, span names and their attributes including the new outcome fields), working APL recipes for error digests and org attribution, prod-DB and PostHog access notes, and the deploy-canary procedure. - e2e/AGENTS.md: the Telemetry service — scenarios can assert on the spans the target actually exported, with the arrival-polling and fixture-tagging gotchas.
Unified · Split
| @@ -0,0 +1,114 @@ | |||
| 1 | --- | ||
| 2 | name: prod-telemetry | ||
| 3 | description: Query Executor's production telemetry — Axiom traces (executor-cloud dataset), prod Postgres via PlanetScale, PostHog product analytics — through the Executor MCP. Use when investigating prod errors, latency, usage, churn signals, or verifying a deploy's telemetry; includes the dataset field layout, working APL recipes, and the error-attribution join. | ||
| 4 | --- | ||
| 5 | |||
| 6 | # Production telemetry access | ||
| 7 | |||
| 8 | All three stores are queryable through the Executor MCP's connected | ||
| 9 | integrations — no dashboards or credentials needed. Verify the connection | ||
| 10 | exists with `connections.list` if a call fails. | ||
| 11 | |||
| 12 | ## Axiom traces (`axiom_mcp`) | ||
| 13 | |||
| 14 | Tool: `axiom_mcp.user.axiomMcpOAuth.querydataset` — the argument is `apl` | ||
| 15 | (NOT `query`). Dataset: `['executor-cloud']` (worker spans; browser spans | ||
| 16 | join the same traces via traceparent). | ||
| 17 | |||
| 18 | **Field layout (the part you'd otherwise rediscover by failed queries):** | ||
| 19 | |||
| 20 | - Custom span attributes live under the JSON map `['attributes.custom']`, | ||
| 21 | NOT as top-level `attributes.*` columns. Read with | ||
| 22 | `['attributes.custom']['mcp.tool.name']`. A nonexistent top-level field is | ||
| 23 | a hard query error ("invalid field"), not an empty result. | ||
| 24 | - Span status: `['status.code']` (`"OK"`/`"ERROR"`), `['status.message']`. | ||
| 25 | - Exceptions: the `events` column carries `exception.type` / | ||
| 26 | `exception.stacktrace` JSON. | ||
| 27 | - OTel basics are top-level: `name`, `trace_id`, `span_id`, | ||
| 28 | `parent_span_id`, `duration`, `_time`. | ||
| 29 | |||
| 30 | **Span names worth querying** (and their custom attrs): | ||
| 31 | |||
| 32 | - `executor.tool.execute` — `mcp.tool.name` (full address), and since | ||
| 33 | PR #992: `executor.tool.outcome` (`ok`/`fail`), | ||
| 34 | `executor.tool.error_code`, `executor.tool.error_status`, | ||
| 35 | `executor.tenant`, `executor.subject`. | ||
| 36 | - `mcp.tool.dispatch` — `mcp.tool.name` (sandbox path), | ||
| 37 | `mcp.tool.integration`, same outcome attrs. | ||
| 38 | - `plugin.openapi.invoke` — `plugin.openapi.method` / `path_template` / | ||
| 39 | `base_url`, and since PR #992 `http.status_code`. | ||
| 40 | - `mcp.request` (outer) — `mcp.auth.organization_id`, | ||
| 41 | `mcp.auth.account_id`, `mcp.tool.name`, CF edge fields (`cf.country`…), | ||
| 42 | MCP client fingerprint (`mcp.client.name`…). | ||
| 43 | |||
| 44 | **Recipe — error signatures by class (the daily-digest query):** | ||
| 45 | |||
| 46 | ```apl | ||
| 47 | ['executor-cloud'] | ||
| 48 | | where _time > ago(1d) | ||
| 49 | | where ['status.code'] == "ERROR" and name == "executor.tool.execute" | ||
| 50 | | extend msg = substring(tostring(['status.message']), 0, 120) | ||
| 51 | | extend tool = tostring(['attributes.custom']['mcp.tool.name']) | ||
| 52 | | summarize n = count() by msg, tool | ||
| 53 | | sort by n desc | ||
| 54 | ``` | ||
| 55 | |||
| 56 | **Recipe — attribute errors to orgs.** Tool spans now carry | ||
| 57 | `executor.tenant` directly (post-#992). For spans from BEFORE that deploy, | ||
| 58 | join through the outer request span: | ||
| 59 | |||
| 60 | ```apl | ||
| 61 | ['executor-cloud'] | ||
| 62 | | where name == "mcp.request" and isnotnull(['attributes.custom']['mcp.auth.organization_id']) | ||
| 63 | | project trace_id, org = tostring(['attributes.custom']['mcp.auth.organization_id']) | ||
| 64 | | join kind=inner ( | ||
| 65 | ['executor-cloud'] | ||
| 66 | | where ['status.code'] == "ERROR" and name == "executor.tool.execute" | ||
| 67 | | project trace_id, msg = substring(tostring(['status.message']), 0, 60) | ||
| 68 | ) on trace_id | ||
| 69 | | summarize n = count() by org, msg | sort by n desc | ||
| 70 | ``` | ||
| 71 | |||
| 72 | **Recipe — upstream failure rate per integration (post-#992 attrs):** | ||
| 73 | |||
| 74 | ```apl | ||
| 75 | ['executor-cloud'] | ||
| 76 | | where _time > ago(1d) and name == "mcp.tool.dispatch" | ||
| 77 | | extend outcome = tostring(['attributes.custom']['executor.tool.outcome']) | ||
| 78 | | extend integration = tostring(['attributes.custom']['mcp.tool.integration']) | ||
| 79 | | where isnotnull(outcome) | ||
| 80 | | summarize calls = count(), fails = countif(outcome == "fail") by integration | ||
| 81 | | extend failRate = todouble(fails) / todouble(calls) | ||
| 82 | | sort by fails desc | ||
| 83 | ``` | ||
| 84 | |||
| 85 | **Known signal caveats** (audited 2026-06-12): | ||
| 86 | |||
| 87 | - Pre-#992 spans: `ToolResult.fail` outcomes (upstream 4xx/5xx, auth | ||
| 88 | rejections) are INVISIBLE — they rode the Effect success channel with no | ||
| 89 | span marker. Don't conclude "no errors" from old data. | ||
| 90 | - Many pre-#992 ERROR spans have an EMPTY `status.message` (tagged errors | ||
| 91 | without a message field) — group those by `events` exception.type instead. | ||
| 92 | - `[object Object]` status messages are the pre-#992 formatting bug. | ||
| 93 | |||
| 94 | ## Prod database (`planetscale_mcp`) | ||
| 95 | |||
| 96 | Read tool needs `{organization: "answer-overflow", database: "executor", | ||
| 97 | branch: "main"}`. It returns `ok: true` even when the SQL failed — check the | ||
| 98 | result text for `Error:`. Use for tenant/integration/connection facts that | ||
| 99 | spans don't carry (row sizes, config shapes, counts). | ||
| 100 | |||
| 101 | ## Product analytics (`posthog_api` / `mcp_posthog_com`) | ||
| 102 | |||
| 103 | Browser-side events only (the ~60-event typed catalog, PR #987; server-side | ||
| 104 | events not built). The org-key `posthog_api` connection covers the REST API; | ||
| 105 | the OAuth MCP connection covers the higher-level tools. | ||
| 106 | |||
| 107 | ## Verifying a deploy's telemetry (Layer-0 canary) | ||
| 108 | |||
| 109 | After deploying telemetry changes: run a known-failing tool call against | ||
| 110 | prod, then assert the expected attributes arrive in Axiom within ~1 min. | ||
| 111 | Absence of data looks identical to health — query for the NEW attribute | ||
| 112 | explicitly rather than eyeballing dashboards. The e2e equivalent runs on | ||
| 113 | every suite: `e2e/cloud/telemetry-contract.test.ts` via the `Telemetry` | ||
| 114 | service (motel `/api/spans/search?attr.<key>=<value>`). | ||
| @@ -77,6 +77,31 @@ const r = yield * session.call("execute", { code: "return 1 + 1;" }); · 76 unmodified lines | |||
| 77 | // human-in-the-loop: session.approvePaused(r.text) resumes a paused execution | 77 | // human-in-the-loop: session.approvePaused(r.text) resumes a paused execution |
| 78 | ``` | 78 | ``` |
| 79 | 79 | ||
| 80 | ## Telemetry scenarios (cloud) | ||
| 81 | |||
| 82 | The suite boots a motel OTLP store and points the target's real exporter at | ||
| 83 | it, so a scenario can assert on the spans the server ACTUALLY exported — | ||
| 84 | the layer where "observability silently went dark" bugs live (an attribute | ||
| 85 | stamped on a span the exporter never carries looks identical to health). | ||
| 86 | |||
| 87 | ```ts | ||
| 88 | const telemetry = yield * Telemetry; // skips when motel didn't boot | ||
| 89 | const span = | ||
| 90 | yield * | ||
| 91 | telemetry.expectSpan({ | ||
| 92 | operation: "executor.tool.execute", | ||
| 93 | attributes: { "mcp.tool.name": failAddress }, // exact match, values stringified | ||
| 94 | }); | ||
| 95 | expect(span.span.tags["executor.tool.outcome"]).toBe("fail"); | ||
| 96 | ``` | ||
| 97 | |||
| 98 | - `expectSpan` polls (~20s): exporters batch, so arrival is | ||
| 99 | eventually-consistent — "the span reaches the store, soon" IS the contract. | ||
| 100 | - Spec gotcha for fixtures: give operations explicit `tags` — tool addresses | ||
| 101 | are `group.leaf`, and an untagged op derives its group from the URL path, | ||
| 102 | so `/fail` does NOT produce a `.fail`-suffixed address. | ||
| 103 | - Prior art: `cloud/telemetry-contract.test.ts`. | ||
| 104 | |||
| 80 | ## Running | 105 | ## Running |
| 81 | 106 | ||
| 82 | ```sh | 107 | ```sh |
| @@ -0,0 +1,181 @@ | |||
| 1 | // Cloud: the telemetry contract, end to end. A tool call that hits an | ||
| 2 | // upstream error wall must be visible in the EXPORTED spans — not just | ||
| 3 | // handled gracefully for the caller. This is the regression class where the | ||
| 4 | // product silently goes dark to operators: `ToolResult.fail` rides the | ||
| 5 | // Effect success channel (a healthy-looking span), and an attribute stamped | ||
| 6 | // on the wrong span simply never arrives in the trace store, which looks | ||
| 7 | // identical to health. So the assertion runs against the OTLP store the dev | ||
| 8 | // stack actually exported to (the suite's motel — the same exporter layer | ||
| 9 | // that ships prod spans to Axiom), driving the whole production topology: | ||
| 10 | // HTTP API → execution engine → sandbox → OpenAPI invoke → a real upstream | ||
| 11 | // returning 502 → span batch → OTLP export. | ||
| 12 | // | ||
| 13 | // Pins two regressions found live in prod (2026-06-12): http.status_code was | ||
| 14 | // annotated inside the inner `OpenApi.invoke` span so the `plugin.openapi. | ||
| 15 | // invoke` span queries target carried it on 0 of ~19.5k spans; and failed | ||
| 16 | // tool calls were indistinguishable from successes on `executor.tool.execute`. | ||
| 17 | import { randomBytes } from "node:crypto"; | ||
| 18 | import { createServer } from "node:http"; | ||
| 19 | |||
| 20 | import { expect } from "@effect/vitest"; | ||
| 21 | import { Effect } from "effect"; | ||
| 22 | import { composePluginApi } from "@executor-js/api/server"; | ||
| 23 | import { openApiHttpPlugin } from "@executor-js/plugin-openapi/api"; | ||
| 24 | import { AuthTemplateSlug, ConnectionName, IntegrationSlug } from "@executor-js/sdk/shared"; | ||
| 25 | |||
| 26 | import { scenario } from "../src/scenario"; | ||
| 27 | import { Api, Target, Telemetry } from "../src/services"; | ||
| 28 | |||
| 29 | const api = composePluginApi([openApiHttpPlugin()] as const); | ||
| 30 | |||
| 31 | /** Two operations: /ok answers 200, /fail answers 502 — the success and | ||
| 32 | * expected-upstream-failure outcome classes the telemetry must separate. */ | ||
| 33 | const upstreamSpec = (baseUrl: string): string => | ||
| 34 | JSON.stringify({ | ||
| 35 | openapi: "3.0.3", | ||
| 36 | info: { title: "Telemetry Upstream", version: "1.0.0" }, | ||
| 37 | servers: [{ url: baseUrl }], | ||
| 38 | paths: { | ||
| 39 | "/ok": { | ||
| 40 | get: { | ||
| 41 | operationId: "ok", | ||
| 42 | summary: "Succeeds", | ||
| 43 | tags: ["probe"], | ||
| 44 | responses: { "200": { description: "" } }, | ||
| 45 | }, | ||
| 46 | }, | ||
| 47 | "/fail": { | ||
| 48 | get: { | ||
| 49 | operationId: "fail", | ||
| 50 | summary: "Always 502", | ||
| 51 | tags: ["probe"], | ||
| 52 | responses: { "200": { description: "" } }, | ||
| 53 | }, | ||
| 54 | }, | ||
| 55 | }, | ||
| 56 | }); | ||
| 57 | |||
| 58 | /** A real upstream on 127.0.0.1: /ok → 200 JSON, anything else → 502 JSON. */ | ||
| 59 | const serveUpstream = Effect.acquireRelease( | ||
| 60 | Effect.callback<{ readonly baseUrl: string; readonly close: () => void }>((resume) => { | ||
| 61 | const server = createServer((request, response) => { | ||
| 62 | const ok = request.url?.startsWith("/ok") ?? false; | ||
| 63 | response.writeHead(ok ? 200 : 502, { "content-type": "application/json" }); | ||
| 64 | response.end(ok ? '{"fine":true}' : '{"error":{"message":"bad gateway"}}'); | ||
| 65 | }); | ||
| 66 | server.listen(0, "127.0.0.1", () => { | ||
| 67 | const address = server.address(); | ||
| 68 | const port = typeof address === "object" && address ? address.port : 0; | ||
| 69 | resume( | ||
| 70 | Effect.succeed({ | ||
| 71 | baseUrl: `http://127.0.0.1:${port}`, | ||
| 72 | close: () => { | ||
| 73 | server.close(); | ||
| 74 | server.closeAllConnections(); | ||
| 75 | }, | ||
| 76 | }), | ||
| 77 | ); | ||
| 78 | }); | ||
| 79 | }), | ||
| 80 | (server) => Effect.sync(server.close), | ||
| 81 | ); | ||
| 82 | |||
| 83 | scenario( | ||
| 84 | "Telemetry · a failing tool call is visible in the exported spans", | ||
| 85 | { timeout: 180_000 }, | ||
| 86 | Effect.scoped( | ||
| 87 | Effect.gen(function* () { | ||
| 88 | const target = yield* Target; | ||
| 89 | const { client: apiClient } = yield* Api; | ||
| 90 | const telemetry = yield* Telemetry; | ||
| 91 | const identity = yield* target.newIdentity(); | ||
| 92 | const client = yield* apiClient(api, identity); | ||
| 93 | |||
| 94 | const upstream = yield* serveUpstream; | ||
| 95 | |||
| 96 | // Identifier-safe slug: it becomes a property path in the sandbox code. | ||
| 97 | const slug = IntegrationSlug.make(`telscn${randomBytes(4).toString("hex")}`); | ||
| 98 | yield* client.openapi.addSpec({ | ||
| 99 | payload: { | ||
| 100 | spec: { kind: "blob", value: upstreamSpec(upstream.baseUrl) }, | ||
| 101 | slug, | ||
| 102 | baseUrl: upstream.baseUrl, | ||
| 103 | authenticationTemplate: [ | ||
| 104 | { | ||
| 105 | slug: "apiKey", | ||
| 106 | type: "apiKey", | ||
| 107 | headers: { Authorization: ["Bearer ", { type: "variable", name: "token" }] }, | ||
| 108 | }, | ||
| 109 | ], | ||
| 110 | }, | ||
| 111 | }); | ||
| 112 | yield* client.connections.create({ | ||
| 113 | payload: { | ||
| 114 | owner: "org", | ||
| 115 | name: ConnectionName.make("main"), | ||
| 116 | integration: slug, | ||
| 117 | template: AuthTemplateSlug.make("apiKey"), | ||
| 118 | value: "telemetry-scenario-token", | ||
| 119 | }, | ||
| 120 | }); | ||
| 121 | |||
| 122 | const tools = yield* client.tools.list({ query: {} }); | ||
| 123 | const addressOf = (op: string) => { | ||
| 124 | const tool = tools.find( | ||
| 125 | (entry) => | ||
| 126 | String(entry.integration) === String(slug) && String(entry.address).endsWith(`.${op}`), | ||
| 127 | ); | ||
| 128 | expect(tool, `the ${op} tool is in the catalog`).toBeDefined(); | ||
| 129 | return String(tool!.address); | ||
| 130 | }; | ||
| 131 | const failAddress = addressOf("fail"); | ||
| 132 | const okAddress = addressOf("ok"); | ||
| 133 | |||
| 134 | // Drive both outcome classes through the full production path. The | ||
| 135 | // failing call still completes for the caller — that is exactly why | ||
| 136 | // the exported span is the only place an operator can see it. | ||
| 137 | for (const address of [okAddress, failAddress]) { | ||
| 138 | const execution = yield* client.executions.execute({ | ||
| 139 | payload: { code: `return await ${address}({});` }, | ||
| 140 | }); | ||
| 141 | expect(execution.status, `the ${address} execution completes`).toBe("completed"); | ||
| 142 | } | ||
| 143 | |||
| 144 | // The failure: outcome attributes on the tool span... | ||
| 145 | const failSpan = yield* telemetry.expectSpan({ | ||
| 146 | operation: "executor.tool.execute", | ||
| 147 | attributes: { "mcp.tool.name": failAddress }, | ||
| 148 | }); | ||
| 149 | expect(failSpan.span.tags, "a failed tool call is marked on the exported span").toMatchObject( | ||
| 150 | { | ||
| 151 | "executor.tool.outcome": "fail", | ||
| 152 | "executor.tool.error_code": "upstream_http_error", | ||
| 153 | "executor.tool.error_status": "502", | ||
| 154 | }, | ||
| 155 | ); | ||
| 156 | expect( | ||
| 157 | failSpan.span.tags["executor.tenant"], | ||
| 158 | "the span carries tenant attribution (no trace-join needed to ask 'whose error?')", | ||
| 159 | ).toBeTruthy(); | ||
| 160 | |||
| 161 | // ...and the upstream status on the HTTP span queries actually target. | ||
| 162 | const invokeSpan = yield* telemetry.expectSpan({ | ||
| 163 | operation: "plugin.openapi.invoke", | ||
| 164 | attributes: { "plugin.openapi.base_url": upstream.baseUrl, "http.status_code": "502" }, | ||
| 165 | }); | ||
| 166 | expect( | ||
| 167 | invokeSpan.span.tags["plugin.openapi.method"], | ||
| 168 | "the invoke span names the method", | ||
| 169 | ).toBe("GET"); | ||
| 170 | |||
| 171 | // The success is distinguishable from the failure. | ||
| 172 | const okSpan = yield* telemetry.expectSpan({ | ||
| 173 | operation: "executor.tool.execute", | ||
| 174 | attributes: { "mcp.tool.name": okAddress }, | ||
| 175 | }); | ||
| 176 | expect(okSpan.span.tags["executor.tool.outcome"], "a successful call is marked ok").toBe( | ||
| 177 | "ok", | ||
| 178 | ); | ||
| 179 | }), | ||
| 180 | ), | ||
| 181 | ); | ||
| @@ -28,6 +28,10 @@ export default async function setup(): Promise<(() => Promise<void>) | void> { · 27 unmodified lines | |||
| 28 | 28 | ||
| 29 | // Suite-owned trace store — every run captures distributed traces. | 29 | // Suite-owned trace store — every run captures distributed traces. |
| 30 | const motel = await bootMotel(); | 30 | const motel = await bootMotel(); |
| 31 | // Publish to the test workers (they inherit this process's env): scenarios | ||
| 32 | // that assert on exported spans yield the Telemetry service, which exists | ||
| 33 | // only when this is set. No motel → those scenarios skip, never fail. | ||
| 34 | if (motel) process.env.E2E_MOTEL_URL = motel.url; | ||
| 31 | 35 | ||
| 32 | const publicUrl = `http://127.0.0.1:${ports.E2E_CLOUD_PORT!}`; | 36 | const publicUrl = `http://127.0.0.1:${ports.E2E_CLOUD_PORT!}`; |
| 33 | let booted; | 37 | let booted; |
| @@ -24,6 +24,7 @@ import { makeApiSurface } from "./surfaces/api"; · 23 unmodified lines | |||
| 24 | import { makeBrowserSurface } from "./surfaces/browser"; | 24 | import { makeBrowserSurface } from "./surfaces/browser"; |
| 25 | import { makeCliSurface } from "./surfaces/cli"; | 25 | import { makeCliSurface } from "./surfaces/cli"; |
| 26 | import { makeMcpSurface } from "./surfaces/mcp"; | 26 | import { makeMcpSurface } from "./surfaces/mcp"; |
| 27 | import { makeTelemetrySurface } from "./surfaces/telemetry"; | ||
| 27 | import { completeOAuthConsent, hasOpenCode, makeOpenCodeHome, warmUp } from "./clients/opencode"; | 28 | import { completeOAuthConsent, hasOpenCode, makeOpenCodeHome, warmUp } from "./clients/opencode"; |
| 28 | import { | 29 | import { |
| 29 | Api, | 30 | Api, |
| @@ -35,6 +36,7 @@ import { · 5 unmodified lines | |||
| 35 | Restart, | 36 | Restart, |
| 36 | RunDir, | 37 | RunDir, |
| 37 | Target, | 38 | Target, |
| 39 | Telemetry, | ||
| 38 | TtlControl, | 40 | TtlControl, |
| 39 | } from "./services"; | 41 | } from "./services"; |
| 40 | import { buildManifest } from "./viewer/manifest"; | 42 | import { buildManifest } from "./viewer/manifest"; |
| @@ -62,7 +64,8 @@ type AllServices = · 21 unmodified lines | |||
| 62 | | Billing | 64 | | Billing |
| 63 | | OpenCode | 65 | | OpenCode |
| 64 | | TtlControl | 66 | | TtlControl |
| 65 | | Restart; | 67 | | Restart |
| 68 | | Telemetry; | ||
| 66 | 69 | ||
| 67 | /** | 70 | /** |
| 68 | * What this target on this host can provide. Services beyond the base are | 71 | * What this target on this host can provide. Services beyond the base are |
| @@ -94,6 +97,9 @@ const contextFor = (target: TargetShape, dir: string): Context.Context<AllServic · 25 unmodified lines | |||
| 94 | if (target.restart) { | 97 | if (target.restart) { |
| 95 | context = Context.add(context, Restart, target.restart); | 98 | context = Context.add(context, Restart, target.restart); |
| 96 | } | 99 | } |
| 100 | if (process.env.E2E_MOTEL_URL) { | ||
| 101 | context = Context.add(context, Telemetry, makeTelemetrySurface(process.env.E2E_MOTEL_URL)); | ||
| 102 | } | ||
| 97 | return context; | 103 | return context; |
| 98 | }; | 104 | }; |
| 99 | 105 | ||
| @@ -11,6 +11,7 @@ import type { ApiSurface } from "./surfaces/api"; · 10 unmodified lines | |||
| 11 | import type { BrowserSurface } from "./surfaces/browser"; | 11 | import type { BrowserSurface } from "./surfaces/browser"; |
| 12 | import type { CliSurface } from "./surfaces/cli"; | 12 | import type { CliSurface } from "./surfaces/cli"; |
| 13 | import type { McpSurface } from "./surfaces/mcp"; | 13 | import type { McpSurface } from "./surfaces/mcp"; |
| 14 | import type { TelemetrySurface } from "./surfaces/telemetry"; | ||
| 14 | import type { completeOAuthConsent, makeOpenCodeHome, warmUp } from "./clients/opencode"; | 15 | import type { completeOAuthConsent, makeOpenCodeHome, warmUp } from "./clients/opencode"; |
| 15 | 16 | ||
| 16 | /** The target under test (always provided). */ | 17 | /** The target under test (always provided). */ |
| @@ -34,6 +35,10 @@ export class Mcp extends Context.Service<Mcp, McpSurface>()("e2e/mcp-oauth") {} · 17 unmodified lines | |||
| 34 | /** Marker: billing limits are enforced on this target. */ | 35 | /** Marker: billing limits are enforced on this target. */ |
| 35 | export class Billing extends Context.Service<Billing, true>()("e2e/billing") {} | 36 | export class Billing extends Context.Service<Billing, true>()("e2e/billing") {} |
| 36 | 37 | ||
| 38 | /** Query the suite's OTLP trace store for spans the target actually exported | ||
| 39 | * (present when the suite booted motel — E2E_MOTEL_URL). */ | ||
| 40 | export class Telemetry extends Context.Service<Telemetry, TelemetrySurface>()("e2e/telemetry") {} | ||
| 41 | |||
| 37 | /** The real OpenCode binary, hermetically driveable (present when installed on this host). */ | 42 | /** The real OpenCode binary, hermetically driveable (present when installed on this host). */ |
| 38 | export interface OpenCodeClient { | 43 | export interface OpenCodeClient { |
| 39 | readonly makeHome: typeof makeOpenCodeHome; | 44 | readonly makeHome: typeof makeOpenCodeHome; |
| @@ -0,0 +1,75 @@ | |||
| 1 | // Telemetry surface: query the suite's motel OTLP store for the spans the | ||
| 2 | // target ACTUALLY exported. This is how a scenario asserts the observability | ||
| 3 | // contract end-to-end — not "did the code create a span object" but "did the | ||
| 4 | // span leave the server, with the attributes production queries depend on". | ||
| 5 | // The dangerous regression mode here is silence (an attribute stamped on the | ||
| 6 | // wrong span, an error riding the success channel): absent data looks exactly | ||
| 7 | // like health, so the contract has to be pinned where the data is read. | ||
| 8 | import { Effect, Schedule } from "effect"; | ||
| 9 | |||
| 10 | /** One exported span, as motel's /api/spans/search returns it. `tags` is the | ||
| 11 | * span's attributes with every value stringified. */ | ||
| 12 | export interface ExportedSpan { | ||
| 13 | readonly traceId: string; | ||
| 14 | readonly rootOperationName: string; | ||
| 15 | readonly span: { | ||
| 16 | readonly spanId: string; | ||
| 17 | readonly operationName: string; | ||
| 18 | readonly status: "ok" | "error"; | ||
| 19 | readonly durationMs: number; | ||
| 20 | readonly tags: Readonly<Record<string, string>>; | ||
| 21 | }; | ||
| 22 | } | ||
| 23 | |||
| 24 | export interface SpanQuery { | ||
| 25 | /** Exact-match attribute filters (motel `attr.<key>=<value>`). */ | ||
| 26 | readonly attributes?: Readonly<Record<string, string>>; | ||
| 27 | readonly operation?: string; | ||
| 28 | readonly traceId?: string; | ||
| 29 | } | ||
| 30 | |||
| 31 | export interface TelemetrySurface { | ||
| 32 | /** One-shot search against the trace store. */ | ||
| 33 | readonly searchSpans: (query: SpanQuery) => Effect.Effect<readonly ExportedSpan[], unknown>; | ||
| 34 | /** Search until at least one span matches. Exporters batch (the app | ||
| 35 | * flushes ~1s after the request), so arrival is eventually-consistent — | ||
| 36 | * polling IS the contract: "the span reaches the store, soon". */ | ||
| 37 | readonly expectSpan: (query: SpanQuery) => Effect.Effect<ExportedSpan, unknown>; | ||
| 38 | } | ||
| 39 | |||
| 40 | export const makeTelemetrySurface = (motelUrl: string): TelemetrySurface => { | ||
| 41 | const searchSpans = (query: SpanQuery) => | ||
| 42 | Effect.gen(function* () { | ||
| 43 | const params = new URLSearchParams({ lookback: "15m", limit: "100" }); | ||
| 44 | if (query.operation) params.set("operation", query.operation); | ||
| 45 | if (query.traceId) params.set("traceId", query.traceId); | ||
| 46 | for (const [key, value] of Object.entries(query.attributes ?? {})) { | ||
| 47 | params.set(`attr.${key}`, value); | ||
| 48 | } | ||
| 49 | const response = yield* Effect.promise(() => fetch(`${motelUrl}/api/spans/search?${params}`)); | ||
| 50 | if (!response.ok) { | ||
| 51 | return yield* Effect.fail( | ||
| 52 | `motel span search responded ${response.status}: ${yield* Effect.promise(() => response.text())}`, | ||
| 53 | ); | ||
| 54 | } | ||
| 55 | const body = (yield* Effect.promise(() => response.json())) as { | ||
| 56 | readonly data?: readonly ExportedSpan[]; | ||
| 57 | }; | ||
| 58 | return body.data ?? []; | ||
| 59 | }); | ||
| 60 | |||
| 61 | return { | ||
| 62 | searchSpans, | ||
| 63 | expectSpan: (query) => | ||
| 64 | searchSpans(query).pipe( | ||
| 65 | Effect.filterOrFail( | ||
| 66 | (spans) => spans.length > 0, | ||
| 67 | () => `no exported span matched ${JSON.stringify(query)}`, | ||
| 68 | ), | ||
| 69 | Effect.map((spans) => spans[0]!), | ||
| 70 | // ~20s ceiling (40 × 500ms): BatchSpanProcessor flushes ~1s after the | ||
| 71 | // request and the dev stack drains on waitUntil; slower is a real bug. | ||
| 72 | Effect.retry(Schedule.both(Schedule.spaced("500 millis"), Schedule.recurs(40))), | ||
| 73 | ), | ||
| 74 | }; | ||
| 75 | }; | ||
| @@ -9,6 +9,7 @@ import type { · 8 unmodified lines | |||
| 9 | ToolSchemaView, | 9 | ToolSchemaView, |
| 10 | } from "@executor-js/sdk/core"; | 10 | } from "@executor-js/sdk/core"; |
| 11 | import { | 11 | import { |
| 12 | annotateToolResultOutcome, | ||
| 12 | authToolFailure, | 13 | authToolFailure, |
| 13 | isToolResult, | 14 | isToolResult, |
| 14 | ToolResult, | 15 | ToolResult, |
| @@ -282,6 +283,10 @@ export const makeExecutorToolInvoker = ( · 267 unmodified lines | |||
| 282 | // raw success value and wrapped — keeps the sandbox-facing contract | 283 | // raw success value and wrapped — keeps the sandbox-facing contract |
| 283 | // uniform without forcing every tiny test plugin to import | 284 | // uniform without forcing every tiny test plugin to import |
| 284 | // `ToolResult.ok`. | 285 | // `ToolResult.ok`. |
| 286 | // Expected failures resolve through the success channel, so without the | ||
| 287 | // outcome annotation the dispatch span reads as healthy even when the | ||
| 288 | // caller hit an upstream error or auth wall. | ||
| 289 | yield* annotateToolResultOutcome(result); | ||
| 285 | if (isToolResult(result)) { | 290 | if (isToolResult(result)) { |
| 286 | return result; | 291 | return result; |
| 287 | } | 292 | } |
| @@ -62,4 +62,10 @@ export class ElicitationDeclinedError extends Schema.TaggedErrorClass<Elicitatio · 61 unmodified lines | |||
| 62 | address: ToolAddress, | 62 | address: ToolAddress, |
| 63 | action: Schema.Literals(["decline", "cancel"]), | 63 | action: Schema.Literals(["decline", "cancel"]), |
| 64 | }, | 64 | }, |
| 65 | ) {} | 65 | ) { |
| 66 | // Derived message so telemetry (span status, logs) labels the failure | ||
| 67 | // instead of rendering an Error with an empty message. | ||
| 68 | override get message(): string { | ||
| 69 | return `Tool approval ${this.action === "cancel" ? "cancelled" : "declined"}: ${this.address}`; | ||
| 70 | } | ||
| 71 | } | ||
| @@ -13,13 +13,24 @@ import { ConnectionName, IntegrationSlug, Owner, ProviderKey, ToolAddress } from · 12 unmodified lines | |||
| 13 | // Tool lifecycle | 13 | // Tool lifecycle |
| 14 | // --------------------------------------------------------------------------- | 14 | // --------------------------------------------------------------------------- |
| 15 | 15 | ||
| 16 | /* Tagged errors without an explicit `message` field define a `message` getter: | ||
| 17 | * `Schema.TaggedErrorClass` instances are real Errors with `message: ""`, and | ||
| 18 | * an empty message propagates everywhere errors are rendered — span | ||
| 19 | * status.message in the tracer, Cause.pretty output, log lines — leaving the | ||
| 20 | * failure unlabeled in telemetry. The getter is derived from the schema fields | ||
| 21 | * (not an own property), so encoding/serialization is unaffected. */ | ||
| 22 | |||
| 16 | export class ToolNotFoundError extends Schema.TaggedErrorClass<ToolNotFoundError>()( | 23 | export class ToolNotFoundError extends Schema.TaggedErrorClass<ToolNotFoundError>()( |
| 17 | "ToolNotFoundError", | 24 | "ToolNotFoundError", |
| 18 | { | 25 | { |
| 19 | address: ToolAddress, | 26 | address: ToolAddress, |
| 20 | suggestions: Schema.optional(Schema.Array(ToolAddress)), | 27 | suggestions: Schema.optional(Schema.Array(ToolAddress)), |
| 21 | }, | 28 | }, |
| 22 | ) {} | 29 | ) { |
| 30 | override get message(): string { | ||
| 31 | return `Tool not found: ${this.address}`; | ||
| 32 | } | ||
| 33 | } | ||
| 23 | 34 | ||
| 24 | export class ToolInvocationError extends Schema.TaggedErrorClass<ToolInvocationError>()( | 35 | export class ToolInvocationError extends Schema.TaggedErrorClass<ToolInvocationError>()( |
| 25 | "ToolInvocationError", | 36 | "ToolInvocationError", |
| @@ -38,7 +49,11 @@ export class ToolBlockedError extends Schema.TaggedErrorClass<ToolBlockedError>( · 12 unmodified lines | |||
| 38 | address: ToolAddress, | 49 | address: ToolAddress, |
| 39 | pattern: Schema.String, | 50 | pattern: Schema.String, |
| 40 | }, | 51 | }, |
| 41 | ) {} | 52 | ) { |
| 53 | override get message(): string { | ||
| 54 | return `Tool blocked by policy "${this.pattern}": ${this.address}`; | ||
| 55 | } | ||
| 56 | } | ||
| 42 | 57 | ||
| 43 | /** Tool row exists but its owning plugin isn't loaded in this executor config. */ | 58 | /** Tool row exists but its owning plugin isn't loaded in this executor config. */ |
| 44 | export class PluginNotLoadedError extends Schema.TaggedErrorClass<PluginNotLoadedError>()( | 59 | export class PluginNotLoadedError extends Schema.TaggedErrorClass<PluginNotLoadedError>()( |
| @@ -47,13 +62,21 @@ export class PluginNotLoadedError extends Schema.TaggedErrorClass<PluginNotLoade · 2 unmodified lines | |||
| 47 | address: ToolAddress, | 62 | address: ToolAddress, |
| 48 | pluginId: Schema.String, | 63 | pluginId: Schema.String, |
| 49 | }, | 64 | }, |
| 50 | ) {} | 65 | ) { |
| 66 | override get message(): string { | ||
| 67 | return `Plugin "${this.pluginId}" is not loaded for tool: ${this.address}`; | ||
| 68 | } | ||
| 69 | } | ||
| 51 | 70 | ||
| 52 | /** Tool was found but its owning plugin has no `invokeTool` handler. */ | 71 | /** Tool was found but its owning plugin has no `invokeTool` handler. */ |
| 53 | export class NoHandlerError extends Schema.TaggedErrorClass<NoHandlerError>()("NoHandlerError", { | 72 | export class NoHandlerError extends Schema.TaggedErrorClass<NoHandlerError>()("NoHandlerError", { |
| 54 | address: ToolAddress, | 73 | address: ToolAddress, |
| 55 | pluginId: Schema.String, | 74 | pluginId: Schema.String, |
| 56 | }) {} | 75 | }) { |
| 76 | override get message(): string { | ||
| 77 | return `Plugin "${this.pluginId}" has no invokeTool handler for tool: ${this.address}`; | ||
| 78 | } | ||
| 79 | } | ||
| 57 | 80 | ||
| 58 | // --------------------------------------------------------------------------- | 81 | // --------------------------------------------------------------------------- |
| 59 | // Integration / connection lifecycle | 82 | // Integration / connection lifecycle |
| @@ -62,7 +85,11 @@ export class NoHandlerError extends Schema.TaggedErrorClass<NoHandlerError>()("N · 2 unmodified lines | |||
| 62 | export class IntegrationNotFoundError extends Schema.TaggedErrorClass<IntegrationNotFoundError>()( | 85 | export class IntegrationNotFoundError extends Schema.TaggedErrorClass<IntegrationNotFoundError>()( |
| 63 | "IntegrationNotFoundError", | 86 | "IntegrationNotFoundError", |
| 64 | { slug: IntegrationSlug }, | 87 | { slug: IntegrationSlug }, |
| 65 | ) {} | 88 | ) { |
| 89 | override get message(): string { | ||
| 90 | return `Integration not found: ${this.slug}`; | ||
| 91 | } | ||
| 92 | } | ||
| 66 | 93 | ||
| 67 | /** An "add integration" operation targeted a slug (namespace) that is already | 94 | /** An "add integration" operation targeted a slug (namespace) that is already |
| 68 | * registered. The core `integrations.register` primitive upserts by design | 95 | * registered. The core `integrations.register` primitive upserts by design |
| @@ -73,14 +100,22 @@ export class IntegrationAlreadyExistsError extends Schema.TaggedErrorClass<Integ · 4 unmodified lines | |||
| 73 | "IntegrationAlreadyExistsError", | 100 | "IntegrationAlreadyExistsError", |
| 74 | { slug: IntegrationSlug }, | 101 | { slug: IntegrationSlug }, |
| 75 | { httpApiStatus: 409 }, | 102 | { httpApiStatus: 409 }, |
| 76 | ) {} | 103 | ) { |
| 104 | override get message(): string { | ||
| 105 | return `Integration already exists: ${this.slug}`; | ||
| 106 | } | ||
| 107 | } | ||
| 77 | 108 | ||
| 78 | /** `integrations.remove` was called on an integration declared statically by a | 109 | /** `integrations.remove` was called on an integration declared statically by a |
| 79 | * plugin at startup (`canRemove: false`). */ | 110 | * plugin at startup (`canRemove: false`). */ |
| 80 | export class IntegrationRemovalNotAllowedError extends Schema.TaggedErrorClass<IntegrationRemovalNotAllowedError>()( | 111 | export class IntegrationRemovalNotAllowedError extends Schema.TaggedErrorClass<IntegrationRemovalNotAllowedError>()( |
| 81 | "IntegrationRemovalNotAllowedError", | 112 | "IntegrationRemovalNotAllowedError", |
| 82 | { slug: IntegrationSlug }, | 113 | { slug: IntegrationSlug }, |
| 83 | ) {} | 114 | ) { |
| 115 | override get message(): string { | ||
| 116 | return `Integration cannot be removed (declared statically by a plugin): ${this.slug}`; | ||
| 117 | } | ||
| 118 | } | ||
| 84 | 119 | ||
| 85 | export class ConnectionNotFoundError extends Schema.TaggedErrorClass<ConnectionNotFoundError>()( | 120 | export class ConnectionNotFoundError extends Schema.TaggedErrorClass<ConnectionNotFoundError>()( |
| 86 | "ConnectionNotFoundError", | 121 | "ConnectionNotFoundError", |
| @@ -89,7 +124,11 @@ export class ConnectionNotFoundError extends Schema.TaggedErrorClass<ConnectionN · 2 unmodified lines | |||
| 89 | integration: IntegrationSlug, | 124 | integration: IntegrationSlug, |
| 90 | name: ConnectionName, | 125 | name: ConnectionName, |
| 91 | }, | 126 | }, |
| 92 | ) {} | 127 | ) { |
| 128 | override get message(): string { | ||
| 129 | return `Connection not found: ${this.integration}.${this.owner}.${this.name}`; | ||
| 130 | } | ||
| 131 | } | ||
| 93 | 132 | ||
| 94 | /** A connection create request was rejected before anything was written: the | 133 | /** A connection create request was rejected before anything was written: the |
| 95 | * input is structurally invalid (no credential inputs for a credentialed | 134 | * input is structurally invalid (no credential inputs for a credentialed |
| @@ -106,7 +145,11 @@ export class InvalidConnectionInputError extends Schema.TaggedErrorClass<Invalid · 10 unmodified lines | |||
| 106 | export class CredentialProviderNotRegisteredError extends Schema.TaggedErrorClass<CredentialProviderNotRegisteredError>()( | 145 | export class CredentialProviderNotRegisteredError extends Schema.TaggedErrorClass<CredentialProviderNotRegisteredError>()( |
| 107 | "CredentialProviderNotRegisteredError", | 146 | "CredentialProviderNotRegisteredError", |
| 108 | { provider: ProviderKey }, | 147 | { provider: ProviderKey }, |
| 109 | ) {} | 148 | ) { |
| 149 | override get message(): string { | ||
| 150 | return `Credential provider not registered: ${this.provider}`; | ||
| 151 | } | ||
| 152 | } | ||
| 110 | 153 | ||
| 111 | /** A connection's value could not be resolved — the provider returned nothing, | 154 | /** A connection's value could not be resolved — the provider returned nothing, |
| 112 | * or an OAuth token refresh failed and the user must re-auth. */ | 155 | * or an OAuth token refresh failed and the user must re-auth. */ |
| @@ -1,4 +1,4 @@ | |||
| 1 | import { Effect, Layer, Option, Predicate, Schema } from "effect"; | 1 | import { Effect, Inspectable, Layer, Option, Predicate, Schema } from "effect"; |
| 2 | import { FetchHttpClient, type HttpClient } from "effect/unstable/http"; | 2 | import { FetchHttpClient, type HttpClient } from "effect/unstable/http"; |
| 3 | import { fumadb } from "@executor-js/fumadb"; | 3 | import { fumadb } from "@executor-js/fumadb"; |
| 4 | import { memoryAdapter } from "@executor-js/fumadb/adapters/memory"; | 4 | import { memoryAdapter } from "@executor-js/fumadb/adapters/memory"; |
| @@ -139,6 +139,7 @@ import { · 134 unmodified lines | |||
| 139 | type OAuthEndpointUrlPolicy, | 139 | type OAuthEndpointUrlPolicy, |
| 140 | } from "./oauth-helpers"; | 140 | } from "./oauth-helpers"; |
| 141 | import { connectionIdentifier } from "./connection-name-identifier"; | 141 | import { connectionIdentifier } from "./connection-name-identifier"; |
| 142 | import { annotateToolResultOutcome } from "./tool-result"; | ||
| 142 | 143 | ||
| 143 | const PLUGIN_STORAGE_DELETE_KEY_BATCH_SIZE = 90; | 144 | const PLUGIN_STORAGE_DELETE_KEY_BATCH_SIZE = 90; |
| 144 | const MAX_APPROVAL_ARGUMENT_PREVIEW_CHARS = 4_000; | 145 | const MAX_APPROVAL_ARGUMENT_PREVIEW_CHARS = 4_000; |
| @@ -2719,10 +2720,20 @@ export const createExecutor = <const TPlugins extends readonly AnyPlugin[] = rea · 2574 unmodified lines | |||
| 2719 | ): Effect.Effect<unknown, ExecuteError> => { | 2720 | ): Effect.Effect<unknown, ExecuteError> => { |
| 2720 | const handler = pickHandler(options); | 2721 | const handler = pickHandler(options); |
| 2721 | return Effect.gen(function* () { | 2722 | return Effect.gen(function* () { |
| 2723 | // oxlint-disable executor/no-instanceof-error, executor/no-unknown-error-message, executor/no-manual-tag-check -- boundary: normalize arbitrary unknown plugin failures into a human-readable message for ToolInvocationError/telemetry | ||
| 2722 | const formatInvocationCauseMessage = (cause: unknown): string => { | 2724 | const formatInvocationCauseMessage = (cause: unknown): string => { |
| 2723 | // oxlint-disable-next-line executor/no-instanceof-error, executor/no-unknown-error-message -- boundary: preserve public execute error message wrapping for unknown plugin failures | 2725 | if (cause instanceof Error && cause.message.length > 0) return cause.message; |
| 2724 | return cause instanceof Error ? cause.message : String(cause); | 2726 | // Non-Error / empty-message causes: `String(plainObject)` renders |
| 2727 | // "[object Object]", which is what telemetry then shows as the only | ||
| 2728 | // label for the failure. Prefer the tag, else stringify structurally. | ||
| 2729 | if (typeof cause === "object" && cause !== null) { | ||
| 2730 | const tag = (cause as { readonly _tag?: unknown })._tag; | ||
| 2731 | if (typeof tag === "string") return tag; | ||
| 2732 | return Inspectable.toStringUnknown(cause, 0); | ||
| 2733 | } | ||
| 2734 | return String(cause); | ||
| 2725 | }; | 2735 | }; |
| 2736 | // oxlint-enable executor/no-instanceof-error, executor/no-unknown-error-message, executor/no-manual-tag-check | ||
| 2726 | const wrapInvocationError = <A, E>( | 2737 | const wrapInvocationError = <A, E>( |
| 2727 | effect: Effect.Effect<A, E>, | 2738 | effect: Effect.Effect<A, E>, |
| 2728 | ): Effect.Effect<A, ToolInvocationError> => | 2739 | ): Effect.Effect<A, ToolInvocationError> => |
| @@ -2877,8 +2888,18 @@ export const createExecutor = <const TPlugins extends readonly AnyPlugin[] = rea · 148 unmodified lines | |||
| 2877 | }), | 2888 | }), |
| 2878 | ); | 2889 | ); |
| 2879 | }).pipe( | 2890 | }).pipe( |
| 2891 | // Expected tool failures (`ToolResult.fail`) resolve through the | ||
| 2892 | // success channel, so the tracer alone would record them as healthy | ||
| 2893 | // spans. Stamp the outcome + error code so telemetry can distinguish | ||
| 2894 | // "tool ran fine" from "user hit an upstream error / auth wall" | ||
| 2895 | // without parsing response bodies. | ||
| 2896 | Effect.tap(annotateToolResultOutcome), | ||
| 2880 | Effect.withSpan("executor.tool.execute", { | 2897 | Effect.withSpan("executor.tool.execute", { |
| 2881 | attributes: { "mcp.tool.name": String(address) }, | 2898 | attributes: { |
| 2899 | "mcp.tool.name": String(address), | ||
| 2900 | "executor.tenant": tenant, | ||
| 2901 | ...(subject != null ? { "executor.subject": subject } : {}), | ||
| 2902 | }, | ||
| 2882 | }), | 2903 | }), |
| 2883 | ); | 2904 | ); |
| 2884 | }; | 2905 | }; |
| @@ -324,7 +324,13 @@ export { buildToolTypeScriptPreview } from "./schema-types"; · 323 unmodified lines | |||
| 324 | export { InternalError } from "./api-errors"; | 324 | export { InternalError } from "./api-errors"; |
| 325 | 325 | ||
| 326 | // ToolResult — typed value-based discriminated union for tool outcomes. | 326 | // ToolResult — typed value-based discriminated union for tool outcomes. |
| 327 | export { ToolResult, isToolResult, type ToolError, type ToolHttpMeta } from "./tool-result"; | 327 | export { |
| 328 | ToolResult, | ||
| 329 | annotateToolResultOutcome, | ||
| 330 | isToolResult, | ||
| 331 | type ToolError, | ||
| 332 | type ToolHttpMeta, | ||
| 333 | } from "./tool-result"; | ||
| 328 | 334 | ||
| 329 | // Stamped boot-time data-migration ledger for the libSQL-backed apps. | 335 | // Stamped boot-time data-migration ledger for the libSQL-backed apps. |
| 330 | export { | 336 | export { |
| @@ -5,7 +5,7 @@ · 4 unmodified lines | |||
| 5 | // the Effect failure channel. | 5 | // the Effect failure channel. |
| 6 | // --------------------------------------------------------------------------- | 6 | // --------------------------------------------------------------------------- |
| 7 | 7 | ||
| 8 | import { Schema } from "effect"; | 8 | import { Effect, Schema } from "effect"; |
| 9 | 9 | ||
| 10 | export const ToolErrorSchema = Schema.Struct({ | 10 | export const ToolErrorSchema = Schema.Struct({ |
| 11 | code: Schema.String, | 11 | code: Schema.String, |
| @@ -56,3 +56,30 @@ const isUnknownToolResult = Schema.is(ToolResultSchema); · 44 unmodified lines | |||
| 56 | 56 | ||
| 57 | export const isToolResult = (value: unknown): value is ToolResult<unknown> => | 57 | export const isToolResult = (value: unknown): value is ToolResult<unknown> => |
| 58 | isUnknownToolResult(value); | 58 | isUnknownToolResult(value); |
| 59 | |||
| 60 | /** | ||
| 61 | * Annotate the current span with the outcome of a tool invocation. | ||
| 62 | * | ||
| 63 | * `ToolResult.fail` rides the Effect *success* channel by design (expected | ||
| 64 | * failures are values, not defects), which means the tracer records those | ||
| 65 | * spans as healthy. Without this, "user keeps hitting 4xx walls" is invisible | ||
| 66 | * to telemetry — the exact class of signal that lets us catch product issues | ||
| 67 | * before they're reported. Stamped attributes: | ||
| 68 | * | ||
| 69 | * - `executor.tool.outcome` — "ok" | "fail" (always, on ToolResults) | ||
| 70 | * - `executor.tool.error_code` — ToolError.code (fail only) | ||
| 71 | * - `executor.tool.error_status` — upstream HTTP status (fail, when present) | ||
| 72 | * | ||
| 73 | * Codes/statuses are enumerable identifiers, never user content — safe span | ||
| 74 | * attributes. Non-ToolResult values (raw success payloads) annotate "ok". | ||
| 75 | */ | ||
| 76 | export const annotateToolResultOutcome = (value: unknown): Effect.Effect<void> => { | ||
| 77 | if (isToolResult(value) && !value.ok) { | ||
| 78 | return Effect.annotateCurrentSpan({ | ||
| 79 | "executor.tool.outcome": "fail", | ||
| 80 | "executor.tool.error_code": value.error.code, | ||
| 81 | ...(value.error.status != null ? { "executor.tool.error_status": value.error.status } : {}), | ||
| 82 | }); | ||
| 83 | } | ||
| 84 | return Effect.annotateCurrentSpan({ "executor.tool.outcome": "ok" }); | ||
| 85 | }; | ||
| @@ -625,6 +625,11 @@ export const invokeWithLayer = ( · 624 unmodified lines | |||
| 625 | 625 | ||
| 626 | return invoke(operation, args, resolvedHeaders, sourceQueryParams).pipe( | 626 | return invoke(operation, args, resolvedHeaders, sourceQueryParams).pipe( |
| 627 | Effect.provide(clientWithBaseUrl), | 627 | Effect.provide(clientWithBaseUrl), |
| 628 | // `invoke` annotates http.status_code on ITS span (`OpenApi.invoke`, | ||
| 629 | // via Effect.fn) — annotateCurrentSpan inside it never reaches this | ||
| 630 | // wrapper span. Stamp the status here too so queries against | ||
| 631 | // `plugin.openapi.invoke` see the upstream outcome directly. | ||
| 632 | Effect.tap((result) => Effect.annotateCurrentSpan({ "http.status_code": result.status })), | ||
| 628 | Effect.withSpan("plugin.openapi.invoke", { | 633 | Effect.withSpan("plugin.openapi.invoke", { |
| 629 | attributes: { | 634 | attributes: { |
| 630 | "plugin.openapi.method": operation.method.toUpperCase(), | 635 | "plugin.openapi.method": operation.method.toUpperCase(), |