History

Route tool defects opaquely; remove legacy envelope shim (#826)

* feat(tool-result): add ToolResult union and dual-shape invoker Adds `ToolResult<T>` / `ToolError` to @executor-js/sdk and a `ToolResult.ok` / `ToolResult.fail` constructor pair. Updates the sandbox tool dispatcher to accept both the new typed union (passes through unchanged) and the legacy `{ data, error }` envelope. The legacy branch now walks known upstream shapes (Microsoft Graph, DealCloud, JSON:API, plain message-bearing bodies) before falling back to a clamped JSON.stringify of the body, so structured 4xx payloads no longer collapse to 'Tool execution failed'. The cause/defect branch keeps its strict .message-only discipline. * feat(plugins): emit ToolResult from openapi, graphql, mcp, google-discovery Migrates each dynamic plugin's invokeTool to wrap its result in the typed ToolResult<T> union. OpenAPI and Google Discovery map non-2xx responses to ToolResult.fail with an extracted upstream message and the raw body in error.details; 2xx responses to ToolResult.ok with { status, headers, data }. GraphQL maps 200-with-errors bodies to ToolResult.fail({ code: 'graphql_errors', ... }) and bare data to ToolResult.ok. MCP maps { isError: true } to ToolResult.fail and content arrays to ToolResult.ok. Static plugin tools (executor.openapi.previewSpec / addSource, executor.graphql.addSource) now also wrap their plain values in ToolResult.ok for shape consistency. Updates plugin tests that asserted on the old { status, headers, data, error } envelope or the bare MCP { content } shape. Tests that consume the OpenAPI plugin's invocation result use the new unwrapInvocation helper from @executor-js/plugin-openapi/testing. * feat(tool-result): make ToolResult mandatory and route defects opaquely Removes the legacy { data, error } envelope shim from the sandbox tool dispatcher. The invoker now passes ToolResult<T> through unchanged and wraps any other plain-value plugin return in ToolResult.ok so the sandbox surface is uniform. Plugin/infra defects no longer pass their message into the sandbox. The dispatcher generates a short hex correlation id, logs the full cause with that id under executor.correlation_id, and rejects with `Internal tool error [<corrId>]`. The QuickJS bridge and the dynamic-worker module template defensively re-stamp the same opaque shape; the MCP host server's top-level execute failure path does the same. ExecutionToolError in-band messages from the execution package's built-in validators (tools.search arg checks, etc.) are still passed through at the QuickJS bridge so model-facing input errors keep their useful diagnostic. Tests: - ToolResult.ok / fail / isToolResult constructor unit tests. - repro tests assert structured upstream payloads now reach the sandbox through ToolResult.error.details (not through .message). - leak tests pin the new invariant: plugin defects only escape as the opaque generic + correlation id; no token / connection string / file path leaks into Error.message. - QuickJS end-to-end defect test confirms the same shape at the sandbox boundary. - Cloud HTTP integration tests, MCP host tests, dynamic-worker invocation tests, and plugin tests updated for the new ToolResult wire shape and the opaque-generic defect contract. * Fix tool result contracts

Rhys Sullivan committed 6 months ago

Commit 278ca7c1 parent 8d986ff35 files · +1322 −340Browse files

Showing the changes to packages/kernel (5 of 35 files). Show all

Unified · Split

5 files
  1. packages/kernel/core/src/strip-types.test.ts +2 −2
  2. packages/kernel/runtime-dynamic-worker/src/invocation.test.ts +33 −9
  3. packages/kernel/runtime-dynamic-worker/src/module-template.ts +9 −1
  4. packages/kernel/runtime-quickjs/src/index.test.ts +46 −0
  5. packages/kernel/runtime-quickjs/src/index.ts +35 −1
packages/kernel/core/src/strip-types.test.ts+2−2
@@ -59,11 +59,11 @@ describe("stripTypeScript", () => { · 58 unmodified lines
5959 // axiom://7bf76f79c5d807272781e9554040aab3 — typed annotation in
6060 // a function expression.
6161 const code = `
62− const fetchDeals = async (sourceId: string): Promise<Array<{ id: string }>> => {
62+ const fetchResources = async (sourceId: string): Promise<Array<{ id: string }>> => {
6363 const result = await tools.executor.sources.list();
6464 return result.items;
6565 };
66− return fetchDeals('dealcloud');
66+ return fetchResources('example-source');
6767 `;
6868 const out = stripTypeScript(code);
6969 expect(out).not.toContain(": string");
packages/kernel/runtime-dynamic-worker/src/invocation.test.ts+33−9
@@ -4,6 +4,7 @@ import * as Cause from "effect/Cause"; · 3 unmodified lines
44 import * as Data from "effect/Data";
55 import * as Effect from "effect/Effect";
66 import type { SandboxToolInvoker } from "@executor-js/codemode-core";
7+import { ExecutionToolError } from "@executor-js/execution";
78 import {
89 ToolDispatcher,
910 makeDynamicWorkerExecutor,
@@ -246,7 +247,7 @@ describe("makeDynamicWorkerExecutor", () => { · 236 unmodified lines
246247 expect(result.result).toBe(7);
247248 });
248249
249− it("surfaces tool errors in execution result", async () => {
250+ it("surfaces infra defects through the worker bridge as an opaque generic", async () => {
250251 const executor = makeDynamicWorkerExecutor({ loader });
251252 const invoker = failingInvoker("not authorized");
252253
@@ -254,7 +255,28 @@ describe("makeDynamicWorkerExecutor", () => { · 1 unmodified line
254255 executor.execute("async () => { return await tools.secret.read({}); }", invoker),
255256 );
256257
257− expect(result.error).toBe("not authorized");
258+ expect(result.error).toBe("Internal tool error");
259+ });
260+
261+ it("preserves public ExecutionToolError messages across the worker bridge", async () => {
262+ const executor = makeDynamicWorkerExecutor({ loader });
263+ const invoker = {
264+ invoke: () =>
265+ Effect.fail(
266+ new ExecutionToolError({
267+ message:
268+ "tools.search expects an object: { query?: string; namespace?: string; limit?: number; offset?: number }",
269+ }),
270+ ),
271+ } satisfies SandboxToolInvoker;
272+
273+ const result = await Effect.runPromise(
274+ executor.execute("async () => await tools.search('github')", invoker),
275+ );
276+
277+ expect(result.error).toBe(
278+ "tools.search expects an object: { query?: string; namespace?: string; limit?: number; offset?: number }",
279+ );
258280 });
259281
260282 it("does not expose host error stack details to sandbox error handlers", async () => {
@@ -284,11 +306,12 @@ describe("makeDynamicWorkerExecutor", () => { · 23 unmodified lines
284306 );
285307
286308 expect(result.error).toBeUndefined();
287− expect(result.result).toMatchObject({ message: "not authorized" });
309+ expect(result.result).toMatchObject({ message: "Internal tool error" });
288310 expect((result.result as { stack?: string }).stack).not.toContain("secret host stack");
311+ expect((result.result as { message?: string }).message).not.toContain("not authorized");
289312 });
290313
291− it("surfaces object-shaped tool errors in execution result", async () => {
314+ it("collapses object-shaped tool defects to an opaque generic", async () => {
292315 const executor = makeDynamicWorkerExecutor({ loader });
293316 const invoker = {
294317 invoke: () =>
@@ -302,11 +325,11 @@ describe("makeDynamicWorkerExecutor", () => { · 7 unmodified lines
302325 executor.execute("async () => { return await tools.secret.read({}); }", invoker),
303326 );
304327
305− expect(result.error).toBe('{"code":"forbidden","detail":"missing team access"}');
328+ expect(result.error).toBe("Internal tool error");
306329 expect(result.result).toBeNull();
307330 });
308331
309− it("surfaces message-bearing object tool errors in execution result", async () => {
332+ it("collapses message-bearing object tool defects to an opaque generic", async () => {
310333 const executor = makeDynamicWorkerExecutor({ loader });
311334 const invoker = {
312335 invoke: () =>
@@ -320,7 +343,8 @@ describe("makeDynamicWorkerExecutor", () => { · 7 unmodified lines
320343 executor.execute("async () => { return await tools.records.query({}); }", invoker),
321344 );
322345
323− expect(result.error).toBe('Field with name "DisplayName" does not exist');
346+ expect(result.error).toBe("Internal tool error");
347+ expect(result.error).not.toContain("DisplayName");
324348 expect(result.result).toBeNull();
325349 });
326350
@@ -366,7 +390,7 @@ describe("makeDynamicWorkerExecutor", () => { · 39 unmodified lines
366390 expect(result.error).toBe("Tool RPC payload contains a circular reference");
367391 });
368392
369− it("returns an execution error for circular tool results", async () => {
393+ it("returns an opaque generic when a tool result can't be serialized", async () => {
370394 const executor = makeDynamicWorkerExecutor({ loader });
371395 const cyclic: Record<string, unknown> = {};
372396 cyclic.self = cyclic;
@@ -377,7 +401,7 @@ describe("makeDynamicWorkerExecutor", () => { · 4 unmodified lines
377401 );
378402
379403 expect(result.result).toBeNull();
380− expect(result.error).toBe("Tool RPC payload contains a circular reference");
404+ expect(result.error).toBe("Internal tool error");
381405 });
382406
383407 it("respects timeout", async () => {
packages/kernel/runtime-dynamic-worker/src/module-template.ts+9−1
@@ -127,6 +127,14 @@ export const buildExecutorModule = (body: string, timeoutMs: number): string => · 126 unmodified lines
127127 " for (const [k, v] of Object.entries(value)) out[k] = __decodeBinary(v, seen);",
128128 " return out;",
129129 " };",
130+ " const __publicToolErrorMessage = (error) => {",
131+ " const values = [error && error.primary, ...(Array.isArray(error && error.failures) ? error.failures : [])];",
132+ " for (const value of values) {",
133+ " if (value && value.__type === 'Error' && value.name === 'ExecutionToolError' && typeof value.message === 'string') return value.message;",
134+ " }",
135+ " if (error && typeof error.message === 'string' && error.message.startsWith('Internal tool error')) return error.message;",
136+ " return null;",
137+ " };",
130138 " const __makeToolsProxy = (path = []) => new Proxy(() => undefined, {",
131139 " get(_target, prop) {",
132140 " if (prop === 'then' || typeof prop === 'symbol') return undefined;",
@@ -138,7 +146,7 @@ export const buildExecutorModule = (body: string, timeoutMs: number): string => · 5 unmodified lines
138146 " return (async () => {",
139147 " const encoded = await __encodeBinary(args[0]);",
140148 " const data = await __dispatcher.call(toolPath, encoded);",
141− " if (!data.ok) throw new Error(data.error && data.error.message ? data.error.message : 'Tool execution failed');",
149+ " if (!data.ok) throw new Error(__publicToolErrorMessage(data.error) || 'Internal tool error');",
142150 " return __decodeBinary(data.result);",
143151 " })();",
144152 " },",
packages/kernel/runtime-quickjs/src/index.test.ts+46−0
@@ -121,6 +121,52 @@ describe("quickjs executor", () => { · 120 unmodified lines
121121 }),
122122 );
123123
124+ it.effect("internal defects reach the sandbox as an opaque generic only", () =>
125+ Effect.gen(function* () {
126+ // Plugin defect carrying sensitive context. The bridge's reject
127+ // path must strip everything except the canonical
128+ // "Internal tool error [<corrId>]" shape — or fall back to the
129+ // bare generic if the upstream invoker hasn't already stamped
130+ // the correlation id (this test exercises the latter path
131+ // because it bypasses makeExecutorToolInvoker).
132+ const invoker: SandboxToolInvoker = {
133+ invoke: () =>
134+ Effect.fail(
135+ Object.assign(
136+ new Error("Authorization: Bearer SECRET_TOKEN_xyz failed against host 10.0.0.5"),
137+ {
138+ stack: "Error\n at /home/svc/executor/packages/plugins/foo:142:11",
139+ },
140+ ) as never,
141+ ),
142+ };
143+
144+ const result = yield* executor.execute(
145+ `
146+ try {
147+ await tools.leaky.call({});
148+ return "should not reach";
149+ } catch (e) {
150+ return e.message;
151+ }
152+ `,
153+ invoker,
154+ );
155+
156+ expect(result.error).toBeUndefined();
157+ const message = String(result.result);
158+ // Either the canonical opaque generic with a correlation id, or
159+ // the bare fallback. Neither must contain any sensitive context.
160+ expect(
161+ message === "Internal tool error" || /^Internal tool error \[[0-9a-f]{8}\]$/.test(message),
162+ ).toBe(true);
163+ expect(message).not.toContain("SECRET_TOKEN_xyz");
164+ expect(message).not.toContain("Authorization");
165+ expect(message).not.toContain("10.0.0.5");
166+ expect(message).not.toContain("packages/plugins");
167+ }),
168+ );
169+
124170 it.effect("handles unknown tool path", () =>
125171 Effect.gen(function* () {
126172 const invoker = makeTestInvoker({});
packages/kernel/runtime-quickjs/src/index.ts+35−1
@@ -47,6 +47,30 @@ const EXECUTION_FILENAME = "executor-quickjs-runtime.js"; · 46 unmodified lines
4747 const toError = (cause: unknown): Error =>
4848 cause instanceof Error ? cause : new Error(String(cause));
4949
50+// Defect surfaced to the sandbox when a tool dispatch reaches the
51+// reject path. Two sources reach here:
52+// - `ExecutionToolError` emitted by the execution package — message
53+// is already either the canonical
54+// `Internal tool error [<corrId>]` opaque generic (plugin defects
55+// passed through `makeExecutorToolInvoker`'s catchCause) or an
56+// intentional public string (validators in engine.ts:
57+// "tools.search expects an object: ...").
58+// - Anything else — assume infra defect and emit a bare opaque
59+// generic.
60+// `_tag === "ExecutionToolError"` is the in-band signal that the
61+// message was generated by code we control and is safe to pass
62+// through; we never read `.cause` / `.stack`.
63+const sandboxDefectMessage = (cause: unknown): string => {
64+ if (cause !== null && typeof cause === "object") {
65+ const tagged = cause as { readonly _tag?: unknown; readonly message?: unknown };
66+ // oxlint-disable-next-line executor/no-manual-tag-check, executor/no-unknown-error-message -- boundary: QuickJS reject path receives an already-rendered ExecutionToolError; we pass its in-band public message through and otherwise emit the opaque generic
67+ if (tagged._tag === "ExecutionToolError" && typeof tagged.message === "string") {
68+ return tagged.message;
69+ }
70+ }
71+ return "Internal tool error";
72+};
73+
5074 const toErrorMessage = (cause: unknown): string => {
5175 if (typeof cause === "object" && cause !== null) {
5276 const message =
@@ -216,7 +240,17 @@ const createToolBridge = ( · 163 unmodified lines
216240 return;
217241 }
218242
219− const errorHandle = context.newError(toErrorMessage(cause));
243+ // The reject path is reserved for true infra defects. The
244+ // upstream tool-invoker has already replaced the message with
245+ // an opaque generic plus a correlation id, but defensively log
246+ // the cause here and emit a stable generic if upstream changes.
247+ const message = sandboxDefectMessage(cause);
248+ try {
249+ console.error("[executor:quickjs] tool dispatch defect", { path, cause });
250+ } catch {
251+ /* ignore logger failures */
252+ }
253+ const errorHandle = context.newError(message);
220254 deferred.reject(errorHandle);
221255 errorHandle.dispose();
222256 },

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t