History

Extract a reusable dev-inference primitive + infer CLI

The eval harness's core capability — drive the real OpenCode binary with the machine's subscription, hermetically, and read structured output — is useful on its own, so pull it out of the eval-specific harness. - e2e/src/clients/inference.ts: runInference({ model, prompt, mcp? }) → { answerText, events, toolNames, ... }. Hermetic OpenCode home with the subscription credential copied in; MCP optional with a pluggable consent strategy. No effect on the developer's own OpenCode state. - e2e/scripts/infer.ts (+ `bun run infer`): the CLI an agent runs while developing — `bun e2e/scripts/infer.ts -m opencode/glm-5.1 "..."`. - evals/harness.ts is now a thin consumer: runTrial = runInference + the selfhost cookie-consent strategy; grading vocabulary unchanged. Also: drop the connect-handoff 'baseUrl resolved' check — running against current main surfaced that integration-level baseUrl is override-only now (per-operation baseUrl from the #968-970 storage refactors), so the check asserted an obsolete field. Documented in EVALS.md. Switch the eval test import to @effect/vitest per the no-vitest-import rule.

Rhys Sullivan committed 5 months ago

Commit 0295eec1 parent c396d0d7 files · +404 −231Browse files

Unified · Split

7 files
  1. e2e/evals/EVALS.md +31 −0
  2. e2e/evals/evals.test.ts +1 −1
  3. e2e/evals/harness.ts +44 −224
  4. e2e/evals/tasks.ts +8 −5
  5. e2e/package.json +2 −1
  6. e2e/scripts/infer.ts +73 −0
  7. e2e/src/clients/inference.ts +245 −0
e2e/evals/EVALS.md+31−0
@@ -6,6 +6,26 @@ ask, does a real agent driving our real MCP server land on the behavior we · 5 unmodified lines
6designed?_ The tool descriptions are the only steering; if a task needs6designed?_ The tool descriptions are the only steering; if a task needs
7system-prompt coaching to pass, that's a finding about the descriptions.7system-prompt coaching to pass, that's a finding about the descriptions.
8 8
9## The inference primitive (reusable on its own)
10
11The eval harness is built on a standalone helper that any agent or test working
12in this repo can use to run real inference, hermetically:
13
14```sh
15# Ask a model a question through the OpenCode subscription — no effect on your
16# own OpenCode history:
17bun e2e/scripts/infer.ts "Reply with exactly: pong"
18bun e2e/scripts/infer.ts -m opencode/glm-5.1 "Summarize this stack trace: ..."
19bun e2e/scripts/infer.ts --json "..." # full JSON event stream
20cd e2e && bun run infer "..." # via the package script
21```
22
23Programmatically, `runInference({ model, prompt, mcp? })` from
24`e2e/src/clients/inference.ts` returns `{ answerText, events, toolNames, ... }`.
25Pass `mcp` to expose one of our MCP servers to the model (with a consent
26strategy); omit it for plain question→answer. The eval harness is just this
27primitive plus grading.
28
9## How it works29## How it works
10 30
11- **Client**: the real OpenCode binary (`opencode run --format json`), in a31- **Client**: the real OpenCode binary (`opencode run --format json`), in a
@@ -126,3 +146,14 @@ description tweaks made because of it): · 114 unmodified lines
126 output/log scrubbing is the pure-upside piece and the cred-hygiene task can146 output/log scrubbing is the pure-upside piece and the cred-hygiene task can
127 A/B any candidate (it already separates "secret in tool call" vs "secret in147 A/B any candidate (it already separates "secret in tool call" vs "secret in
128 answer"). Tracking this as a known gap, not a bug to fix now.148 answer"). Tracking this as a known gap, not a bug to fix now.
149- **2026-06-12 · integration-level baseUrl is now override-only**: against
150 current main, an add-by-spec-URL stores `config.baseUrl: null` even though
151 the spec declares `servers[0].url`. Not a regression — the storage refactors
152 (#968–970) moved the host to PER-OPERATION baseUrl (baked into each compiled
153 tool from the spec's `servers`), so tool calls still reach the right host and
154 integration-level baseUrl became an override-only field. The connect-handoff
155 task's old "baseUrl resolved to the emulator" check asserted the obsolete
156 integration-level field and started failing the moment the eval ran against
157 main instead of the PR #957 branch — exactly the kind of semantic drift the
158 eval exists to surface. Check removed; "registered + apikey derived" already
159 proves the spec compiled with auth.
e2e/evals/evals.test.ts+1−1
@@ -5,7 +5,7 @@ · 4 unmodified lines
5import { mkdirSync, writeFileSync } from "node:fs";5import { mkdirSync, writeFileSync } from "node:fs";
6import { join } from "node:path";6import { join } from "node:path";
7 7
8import { describe, expect, it } from "vitest";8import { describe, expect, it } from "@effect/vitest";
9 9
10import { resolveTarget } from "../targets/registry";10import { resolveTarget } from "../targets/registry";
11import { hasOpenCode } from "../src/clients/opencode";11import { hasOpenCode } from "../src/clients/opencode";
e2e/evals/harness.ts+44−224
@@ -1,27 +1,29 @@
1// The eval harness: run the REAL OpenCode binary with real Go-subscription1// The eval harness: run a real OpenCode inference trial against a real target,
2// inference against a real target, then grade the result with deterministic2// then grade the result with deterministic checks. One eval = one task × one
3// checks. One eval = one task × one model × one trial; the vitest file fans3// model × one trial; the vitest file fans out the matrix and aggregates pass
4// out the matrix and aggregates pass rates (see report.ts).4// rates. The inference mechanics live in the reusable `runInference` primitive
5// (src/clients/inference.ts); this file adds only the target-specific MCP
6// consent strategy and the grading vocabulary.
5//7//
6// Design intent (EVALS.md): the agent gets the user's one-line ask and8// Design intent (EVALS.md): the agent gets the user's one-line ask and
7// whatever our MCP server advertises — no extra system prompt, no coached9// whatever our MCP server advertises — no extra system prompt, no coached
8// tool order. The tool descriptions are what's under test.10// tool order. The tool descriptions are what's under test.
9import { spawn, spawnSync } from "node:child_process";
10import { copyFileSync, existsSync, mkdirSync, writeFileSync } from "node:fs";
11import { homedir } from "node:os";
12import { join } from "node:path";
13
14import { Effect } from "effect";11import { Effect } from "effect";
15 12
16import type { Identity, Target } from "../src/target";13import type { Identity, Target } from "../src/target";
17import { makeOpenCodeHome, warmUp, type OpenCodeHome } from "../src/clients/opencode";14import {
15 hasOpenCodeSubscription,
16 runInference,
17 toolTrafficOf,
18 type InferenceResult,
19} from "../src/clients/inference";
18 20
19// ---------------------------------------------------------------------------21// ---------------------------------------------------------------------------
20// Config — every knob is an env var so CI and local runs share one path.22// Config — every knob is an env var so CI and local runs share one path.
21// ---------------------------------------------------------------------------23// ---------------------------------------------------------------------------
22 24
23export const EVAL_DEFAULT_MODELS = [25export const EVAL_DEFAULT_MODELS = [
24 // Spread across separate Go quota pools; see EVALS.md for the full table.26 // Spread across separate subscription quota pools; see EVALS.md for the table.
25 "opencode/deepseek-v4-flash",27 "opencode/deepseek-v4-flash",
26 "opencode/minimax-m2.5",28 "opencode/minimax-m2.5",
27 "opencode/kimi-k2.5",29 "opencode/kimi-k2.5",
@@ -39,235 +41,53 @@ export const evalTrials = (): number => { · 11 unmodified lines
39 return Number.isInteger(parsed) && parsed > 0 ? parsed : 3;41 return Number.isInteger(parsed) && parsed > 0 ? parsed : 3;
40};42};
41 43
42/** The host machine's Go credential, copied into each hermetic home so the44export const hasGoSubscription = hasOpenCodeSubscription;
43 * throwaway OpenCode can use the subscription. */
44const hostAuthFile = (): string => join(homedir(), ".local", "share", "opencode", "auth.json");
45
46export const hasGoSubscription = (): boolean => existsSync(hostAuthFile());
47
48// ---------------------------------------------------------------------------
49// One trial: spawn `opencode run` headless, collect the JSON event stream.
50// ---------------------------------------------------------------------------
51
52export interface TrialEvent {
53 readonly type: string;
54 readonly part?: {
55 readonly type?: string;
56 readonly text?: string;
57 readonly tool?: string;
58 readonly state?: {
59 readonly status?: string;
60 readonly input?: unknown;
61 readonly output?: unknown;
62 };
63 };
64}
65
66export interface TrialResult {
67 /** Every JSON event opencode emitted, in order. */
68 readonly events: readonly TrialEvent[];
69 /** All assistant text parts joined — "what the user read". */
70 readonly answerText: string;
71 /** Raw stdout (JSONL) for the artifact dir. */
72 readonly rawStdout: string;
73 readonly exitCode: number | null;
74 readonly durationMs: number;
75}
76
77export const trialAnswerText = (events: readonly TrialEvent[]): string =>
78 events
79 .filter((e) => e.type === "text" && typeof e.part?.text === "string")
80 .map((e) => e.part?.text ?? "")
81 .join("\n");
82 45
83/** Tool-call inputs/outputs as strings, for transcript-wide content checks46// Re-exported for grading code (tasks.ts) that inspects tool traffic.
84 * (e.g. "the credential never appears anywhere the model produced"). */47export const trialToolTraffic = toolTrafficOf;
85export const trialToolTraffic = (events: readonly TrialEvent[]): string =>
86 events
87 .filter((e) => e.type === "tool_use")
88 .map((e) => JSON.stringify(e.part?.state ?? {}))
89 .join("\n");
90 48
91/** Names of tools the model invoked, for "used our MCP tools at all" checks. */49export type TrialResult = InferenceResult;
92export const trialToolNames = (events: readonly TrialEvent[]): readonly string[] =>
93 events.filter((e) => e.type === "tool_use").map((e) => e.part?.tool ?? "");
94 50
95export interface RunTrialOptions {51export interface RunTrialOptions {
96 readonly serverName: string;52 readonly serverName: string;
97 readonly mcpUrl: string;53 readonly mcpUrl: string;
98 readonly model: string;54 readonly model: string;
99 readonly prompt: string;55 readonly prompt: string;
100 /** Identity whose email answers the MCP OAuth consent hop. */56 /** Identity whose session cookie answers the MCP OAuth consent hop. */
101 readonly identity: Identity;57 readonly identity: Identity;
102 readonly timeoutMs: number;58 readonly timeoutMs: number;
103}59}
104 60
105/** A hermetic OpenCode home wired for real inference: the target's MCP server61/** Answer OpenCode's recorded browser hop the way a signed-in selfhost user
106 * plus the host's Go credential. Tool permissions are pre-allowed — evals62 * would: drive the authorize URL with the identity's Better Auth session
107 * measure model behavior, not consent dialogs. */63 * cookie and deliver the code to OpenCode's localhost callback. (Cloud's
108const makeEvalHome = (serverName: string, mcpUrl: string): OpenCodeHome => {64 * emulator dialect uses login_hint instead — this is the selfhost path.) */
109 const home = makeOpenCodeHome(serverName, mcpUrl);65const cookieConsent =
110 const authDir = join(home.env.XDG_DATA_HOME ?? "", "opencode");66 (identity: Identity) =>
111 mkdirSync(authDir, { recursive: true });67 async (authorizationUrl: string): Promise<void> => {
112 copyFileSync(hostAuthFile(), join(authDir, "auth.json"));68 const cookie = identity.headers?.cookie ?? "";
113 // Extend the generated opencode.json: keep the MCP server, allow all tools,69 const authorize = await fetch(authorizationUrl, { headers: { cookie }, redirect: "manual" });
114 // disable share/autoupdate noise.70 const location = authorize.headers.get("location");
115 const configPath = join(home.projectDir, "opencode.json");71 if (!location) {
116 writeFileSync(72 throw new Error(`eval consent: authorize did not redirect (${authorize.status})`);
117 configPath,
118 JSON.stringify({
119 $schema: "https://opencode.ai/config.json",
120 autoupdate: false,
121 share: "disabled",
122 permission: { "*": "allow" },
123 mcp: { [serverName]: { type: "remote", url: mcpUrl } },
124 }),
125 );
126 return home;
127};
128
129/** Play the signed-in human for OpenCode's recorded browser hop: sign in for
130 * a Better Auth session cookie, drive the authorize URL with it, and deliver
131 * the resulting code to OpenCode's localhost callback. (The scenario-side
132 * completeOAuthConsent uses login_hint — that's the cloud emulator's dialect;
133 * selfhost's Better Auth consent requires the cookie.) */
134const consentWithCookie = async (
135 home: OpenCodeHome,
136 identity: Identity,
137 baseUrl: string,
138 sinceIndex: number,
139): Promise<void> => {
140 const deadline = Date.now() + 60_000;
141 while (Date.now() < deadline) {
142 const authorizationUrl = home.openedUrls()[sinceIndex];
143 if (authorizationUrl) {
144 const cookie = identity.headers?.cookie ?? "";
145 const authorize = await fetch(authorizationUrl, {
146 headers: { cookie },
147 redirect: "manual",
148 });
149 const location = authorize.headers.get("location");
150 if (!location) {
151 throw new Error(`eval consent: authorize did not redirect (${authorize.status})`);
152 }
153 // Hand the code to OpenCode's local callback server.
154 const callback = await fetch(location);
155 if (!callback.ok) throw new Error(`eval consent: callback failed (${callback.status})`);
156 return;
157 }73 }
158 await new Promise((tick) => setTimeout(tick, 250));74 const callback = await fetch(location);
159 }75 if (!callback.ok) throw new Error(`eval consent: callback failed (${callback.status})`);
160 throw new Error("eval consent: opencode never opened an authorization URL");76 };
161};
162
163/** Connect OpenCode to the target's MCP server before the trial — a user's
164 * OpenCode is already authenticated by the time they ask for work, and
165 * `opencode run` does not initiate MCP OAuth itself (without this, the
166 * executor tools simply never exist and the model free-styles with bash). */
167const preAuthMcp = async (
168 home: OpenCodeHome,
169 serverName: string,
170 identity: Identity,
171 baseUrl: string,
172): Promise<void> => {
173 // First-run database migration in a bare project — `mcp auth` misbehaves
174 // if it doubles as first run (see warmUp's doc comment).
175 warmUp(home);
176 // The auth command must run ASYNC (spawn, not spawnSync): the consent
177 // helper polls on timers, and a blocked event loop would starve it while
178 // `mcp auth` sits waiting for the browser hop it recorded via the shim.
179 const sinceIndex = home.openedUrls().length;
180 const auth = spawn("opencode", ["mcp", "auth", serverName], {
181 cwd: home.projectDir,
182 env: home.env,
183 stdio: ["ignore", "pipe", "pipe"],
184 });
185 const authExit = new Promise<void>((resolve) => {
186 const killer = setTimeout(() => auth.kill("SIGKILL"), 90_000);
187 auth.once("exit", () => {
188 clearTimeout(killer);
189 resolve();
190 });
191 });
192 await consentWithCookie(home, identity, baseUrl, sinceIndex);
193 await authExit;
194 const listed = spawnSync("opencode", ["mcp", "list"], {
195 cwd: home.projectDir,
196 env: home.env,
197 timeout: 60_000,
198 encoding: "utf8",
199 });
200 if (!`${listed.stdout}`.includes("connected")) {
201 throw new Error(`eval pre-auth: MCP server never reached "connected" for ${serverName}`);
202 }
203};
204 77
205export const runTrial = (options: RunTrialOptions): Effect.Effect<TrialResult, Error> =>78export const runTrial = (options: RunTrialOptions): Effect.Effect<TrialResult, Error> =>
206 Effect.promise(async () => {79 Effect.promise(() =>
207 const home = makeEvalHome(options.serverName, options.mcpUrl);80 runInference({
208 const baseUrl = new URL(options.mcpUrl).origin;81 model: options.model,
209 await preAuthMcp(home, options.serverName, options.identity, baseUrl);82 prompt: options.prompt,
210 const startedAt = Date.now();83 timeoutMs: options.timeoutMs,
211 84 mcp: {
212 const child = spawn(85 serverName: options.serverName,
213 "opencode",86 url: options.mcpUrl,
214 ["run", "-m", options.model, "--format", "json", options.prompt],87 consent: cookieConsent(options.identity),
215 {
216 cwd: home.projectDir,
217 // PWD must match cwd: the inherited value points at the eval RUNNER's
218 // checkout, and a leaked path invites the model to wander our repo
219 // instead of acting like a user in an empty project.
220 env: { ...home.env, PWD: home.projectDir },
221 stdio: ["ignore", "pipe", "pipe"],
222 },88 },
223 );89 }),
224 90 );
225 let stdout = "";
226 let stderr = "";
227 child.stdout.on("data", (chunk: Buffer) => (stdout += chunk.toString("utf8")));
228 child.stderr.on("data", (chunk: Buffer) => (stderr += chunk.toString("utf8")));
229
230 // Play the signed-in human whenever OpenCode opens an OAuth consent URL.
231 // Pre-auth already granted the MCP session; this loop stays alive for the
232 // whole trial in case the agent triggers another browser hop mid-run.
233 let consented = home.openedUrls().length;
234 const consentLoop = setInterval(() => {
235 const urls = home.openedUrls();
236 if (urls.length > consented) {
237 const index = consented;
238 consented = urls.length;
239 void consentWithCookie(home, options.identity, baseUrl, index).catch(() => {});
240 }
241 }, 300);
242
243 const exitCode = await new Promise<number | null>((resolve) => {
244 const killer = setTimeout(() => child.kill("SIGKILL"), options.timeoutMs);
245 child.once("exit", (code) => {
246 clearTimeout(killer);
247 resolve(code);
248 });
249 });
250 clearInterval(consentLoop);
251
252 const events: TrialEvent[] = [];
253 for (const line of stdout.split("\n")) {
254 if (!line.trim()) continue;
255 // oxlint-disable-next-line executor/no-try-catch-or-throw, executor/no-json-parse -- boundary: tolerant parse of opencode's JSONL event stream
256 try {
257 events.push(JSON.parse(line) as TrialEvent);
258 } catch {
259 // Non-JSON line (banner, warning) — keep going.
260 }
261 }
262
263 return {
264 events,
265 answerText: trialAnswerText(events),
266 rawStdout: stdout.length > 0 ? stdout : stderr,
267 exitCode,
268 durationMs: Date.now() - startedAt,
269 };
270 });
271 91
272// ---------------------------------------------------------------------------92// ---------------------------------------------------------------------------
273// Task registry — a task is a prompt plus deterministic graders.93// Task registry — a task is a prompt plus deterministic graders.
e2e/evals/tasks.ts+8−5
@@ -50,10 +50,18 @@ const integrationChecks = async ( · 49 unmodified lines
50 ctx: GradeContext,50 ctx: GradeContext,
51 integration: string,51 integration: string,
52): Promise<readonly GradeCheck[]> => {52): Promise<readonly GradeCheck[]> => {
53 // The config endpoint returns non-null only once addSpec's transaction
54 // committed (integration row + operations together), so this doubles as the
55 // "operations compiled" signal.
53 const config = (await ctx56 const config = (await ctx
54 .apiGet(`/api/openapi/integrations/${integration}/config`)57 .apiGet(`/api/openapi/integrations/${integration}/config`)
55 .catch(() => null)) as OpenApiConfig | null;58 .catch(() => null)) as OpenApiConfig | null;
56 const methods = config?.authenticationTemplate ?? [];59 const methods = config?.authenticationTemplate ?? [];
60 // NOTE: we intentionally do NOT assert the integration-level `config.baseUrl`.
61 // The host moved to per-operation baseUrl (baked into each tool at compile
62 // time from the spec's `servers`); integration-level baseUrl is now an
63 // override-only field and is null for a plain add-by-spec. The eval surfaced
64 // this when it started running against current main — see EVALS.md.
57 return [65 return [
58 check("integration registered under the asked-for slug", config !== null),66 check("integration registered under the asked-for slug", config !== null),
59 check(67 check(
@@ -61,11 +69,6 @@ const integrationChecks = async ( · 1 unmodified line
61 methods.some((m) => m.kind === "apikey"),69 methods.some((m) => m.kind === "apikey"),
62 `methods: ${JSON.stringify(methods.map((m) => m.kind))}`,70 `methods: ${JSON.stringify(methods.map((m) => m.kind))}`,
63 ),71 ),
64 check(
65 "baseUrl resolved to the emulator",
66 (config?.baseUrl ?? "").startsWith(EMULATOR_BASE),
67 `baseUrl: ${config?.baseUrl}`,
68 ),
69 ];72 ];
70};73};
71 74
e2e/package.json+2−1
@@ -12,7 +12,8 @@ · 11 unmodified lines
12 "serve": "bun scripts/rebuild-viewer.ts && bun scripts/serve.ts",12 "serve": "bun scripts/rebuild-viewer.ts && bun scripts/serve.ts",
13 "typecheck": "tsc --noEmit",13 "typecheck": "tsc --noEmit",
14 "test:desktop": "vitest run --project desktop",14 "test:desktop": "vitest run --project desktop",
15 "test:evals": "EVAL=1 vitest run --project evals"15 "test:evals": "EVAL=1 vitest run --project evals",
16 "infer": "bun scripts/infer.ts"
16 },17 },
17 "dependencies": {18 "dependencies": {
18 "@executor-js/api": "workspace:*",19 "@executor-js/api": "workspace:*",
e2e/scripts/infer.ts+73−0
@@ -0,0 +1,73 @@
1// Dev-time inference for agents working in this repo: ask a real model a
2// question through the machine's OpenCode subscription, hermetically (no effect
3// on your own OpenCode history). Prints the model's answer to stdout.
4//
5// bun e2e/scripts/infer.ts "What is 6 * 7?"
6// bun e2e/scripts/infer.ts -m opencode/glm-5.1 "Summarize this error: ..."
7// bun e2e/scripts/infer.ts --json "..." # full JSON event stream
8// cd e2e && bun run infer "..." # via the package script
9//
10// Model ids are `opencode/<name>` — see e2e/evals/EVALS.md for the
11// subscription model/quota table. Default is a cheap, high-quota model.
12import { hasOpenCodeSubscription, runInference } from "../src/clients/inference";
13import { hasOpenCode } from "../src/clients/opencode";
14
15const DEFAULT_MODEL = "opencode/deepseek-v4-flash";
16
17const parseArgs = (argv: readonly string[]) => {
18 let model = DEFAULT_MODEL;
19 let json = false;
20 const rest: string[] = [];
21 for (let i = 0; i < argv.length; i++) {
22 const arg = argv[i];
23 if (arg === "-m" || arg === "--model") {
24 model = argv[++i] ?? model;
25 } else if (arg === "--json") {
26 json = true;
27 } else {
28 rest.push(arg);
29 }
30 }
31 return { model, json, prompt: rest.join(" ").trim() };
32};
33
34const main = async () => {
35 if (!hasOpenCode()) {
36 process.stderr.write("infer: the `opencode` binary is not installed.\n");
37 process.exit(2);
38 }
39 if (!hasOpenCodeSubscription()) {
40 process.stderr.write(
41 "infer: no OpenCode credential found (~/.local/share/opencode/auth.json). Run `opencode auth login` first.\n",
42 );
43 process.exit(2);
44 }
45
46 const { model, json, prompt } = parseArgs(process.argv.slice(2));
47 if (!prompt) {
48 process.stderr.write(
49 'infer: no prompt. Usage: bun e2e/scripts/infer.ts [-m model] "your prompt"\n',
50 );
51 process.exit(2);
52 }
53
54 const result = await runInference({ model, prompt });
55 if (json) {
56 process.stdout.write(
57 result.rawStdout.endsWith("\n") ? result.rawStdout : `${result.rawStdout}\n`,
58 );
59 } else {
60 process.stdout.write(`${result.answerText}\n`);
61 }
62 if (result.exitCode !== 0) {
63 process.stderr.write(
64 `infer: opencode exited ${result.exitCode} after ${result.durationMs}ms\n`,
65 );
66 }
67};
68
69// oxlint-disable-next-line executor/no-try-catch-or-throw -- boundary: top-level CLI error handler
70main().catch((error) => {
71 process.stderr.write(`infer: ${String(error)}\n`);
72 process.exit(1);
73});
e2e/src/clients/inference.ts+245−0
@@ -0,0 +1,245 @@
1// Dev-time inference: drive the REAL OpenCode binary as a one-shot model call,
2// hermetically. A throwaway OpenCode home borrows the machine's OpenCode
3// subscription credential, so an agent (or a test) working in this repo can
4// ask a real model a question — optionally with one of our MCP servers wired
5// in — and get structured output back, without touching the developer's own
6// OpenCode state/history.
7//
8// Two consumers:
9// - `e2e/scripts/infer.ts` — the CLI an agent runs while developing.
10// - `e2e/evals/` — the eval harness, which adds grading on top.
11//
12// What OpenCode does with the prompt, the tools, and (when wired) MCP OAuth is
13// entirely its own code; we only provide the credential and read the result.
14import { spawn, spawnSync } from "node:child_process";
15import { copyFileSync, existsSync, mkdirSync, writeFileSync } from "node:fs";
16import { homedir } from "node:os";
17import { join } from "node:path";
18
19import { makeOpenCodeHome, warmUp, type OpenCodeHome } from "./opencode";
20
21// ---------------------------------------------------------------------------
22// Subscription credential
23// ---------------------------------------------------------------------------
24
25/** The machine's OpenCode credential (e.g. the Go subscription), copied into
26 * each hermetic home so the throwaway OpenCode can run inference. */
27const hostAuthFile = (): string => join(homedir(), ".local", "share", "opencode", "auth.json");
28
29export const hasOpenCodeSubscription = (): boolean => existsSync(hostAuthFile());
30
31// ---------------------------------------------------------------------------
32// Event stream
33// ---------------------------------------------------------------------------
34
35export interface InferenceEvent {
36 readonly type: string;
37 readonly part?: {
38 readonly type?: string;
39 readonly text?: string;
40 readonly tool?: string;
41 readonly state?: {
42 readonly status?: string;
43 readonly input?: unknown;
44 readonly output?: unknown;
45 };
46 };
47}
48
49export interface InferenceResult {
50 /** Every JSON event opencode emitted, in order. */
51 readonly events: readonly InferenceEvent[];
52 /** All assistant text parts joined — "what the user read". */
53 readonly answerText: string;
54 /** Names of the tools the model invoked. */
55 readonly toolNames: readonly string[];
56 /** Raw stdout (JSONL), or stderr if stdout was empty — for artifacts. */
57 readonly rawStdout: string;
58 readonly exitCode: number | null;
59 readonly durationMs: number;
60}
61
62export const answerTextOf = (events: readonly InferenceEvent[]): string =>
63 events
64 .filter((e) => e.type === "text" && typeof e.part?.text === "string")
65 .map((e) => e.part?.text ?? "")
66 .join("\n");
67
68/** Tool-call inputs/outputs as one string, for content checks (e.g. "the
69 * credential never appears anywhere the model produced"). */
70export const toolTrafficOf = (events: readonly InferenceEvent[]): string =>
71 events
72 .filter((e) => e.type === "tool_use")
73 .map((e) => JSON.stringify(e.part?.state ?? {}))
74 .join("\n");
75
76export const toolNamesOf = (events: readonly InferenceEvent[]): readonly string[] =>
77 events.filter((e) => e.type === "tool_use").map((e) => e.part?.tool ?? "");
78
79const parseEvents = (stdout: string): InferenceEvent[] => {
80 const events: InferenceEvent[] = [];
81 for (const line of stdout.split("\n")) {
82 if (!line.trim()) continue;
83 // oxlint-disable-next-line executor/no-try-catch-or-throw, executor/no-json-parse -- boundary: tolerant parse of opencode's JSONL event stream
84 try {
85 events.push(JSON.parse(line) as InferenceEvent);
86 } catch {
87 // Non-JSON line (banner, warning) — keep going.
88 }
89 }
90 return events;
91};
92
93// ---------------------------------------------------------------------------
94// Hermetic home
95// ---------------------------------------------------------------------------
96
97/** Optional MCP server to expose to the model. `consent` answers OpenCode's
98 * recorded browser hop: given the authorization URL it opened, deliver the
99 * code to OpenCode's local callback (the strategy is target-specific —
100 * Better Auth cookie, a login_hint redirect, etc.). */
101export interface McpWiring {
102 readonly serverName: string;
103 readonly url: string;
104 readonly consent: (authorizationUrl: string) => Promise<void>;
105}
106
107/** A throwaway OpenCode home with the subscription credential and all tool
108 * permissions pre-allowed (inference measures model behavior, not consent
109 * dialogs). Wires in one MCP server when `mcp` is provided. */
110const makeInferenceHome = (mcp?: McpWiring): OpenCodeHome => {
111 const home = makeOpenCodeHome(mcp?.serverName ?? "none", mcp?.url ?? "http://127.0.0.1:0/mcp");
112 const authDir = join(home.env.XDG_DATA_HOME ?? "", "opencode");
113 mkdirSync(authDir, { recursive: true });
114 copyFileSync(hostAuthFile(), join(authDir, "auth.json"));
115 writeFileSync(
116 join(home.projectDir, "opencode.json"),
117 JSON.stringify({
118 $schema: "https://opencode.ai/config.json",
119 autoupdate: false,
120 share: "disabled",
121 permission: { "*": "allow" },
122 ...(mcp ? { mcp: { [mcp.serverName]: { type: "remote", url: mcp.url } } } : {}),
123 }),
124 );
125 return home;
126};
127
128/** Connect OpenCode to an MCP server before the run — `opencode run` does not
129 * initiate MCP OAuth itself, so without this the server's tools never exist
130 * and the model free-styles with its built-ins. Drives `opencode mcp auth`
131 * and answers the browser hop via `mcp.consent`. */
132const connectMcp = async (home: OpenCodeHome, mcp: McpWiring): Promise<void> => {
133 // First-run DB migration in a bare project — `mcp auth` misbehaves if it
134 // doubles as first run (see warmUp's doc comment).
135 warmUp(home);
136 // ASYNC spawn (not spawnSync): the consent step polls on timers, and a
137 // blocked event loop would starve it while `mcp auth` waits for the browser
138 // hop it records via the open(1) shim.
139 const sinceIndex = home.openedUrls().length;
140 const auth = spawn("opencode", ["mcp", "auth", mcp.serverName], {
141 cwd: home.projectDir,
142 env: home.env,
143 stdio: ["ignore", "pipe", "pipe"],
144 });
145 const authExit = new Promise<void>((resolve) => {
146 const killer = setTimeout(() => auth.kill("SIGKILL"), 90_000);
147 auth.once("exit", () => {
148 clearTimeout(killer);
149 resolve();
150 });
151 });
152 await waitForAuthorizationUrl(home, sinceIndex).then(mcp.consent);
153 await authExit;
154 const listed = spawnSync("opencode", ["mcp", "list"], {
155 cwd: home.projectDir,
156 env: home.env,
157 timeout: 60_000,
158 encoding: "utf8",
159 });
160 if (!`${listed.stdout}`.includes("connected")) {
161 throw new Error(`inference: MCP server "${mcp.serverName}" never reached "connected"`);
162 }
163};
164
165const waitForAuthorizationUrl = async (home: OpenCodeHome, sinceIndex: number): Promise<string> => {
166 const deadline = Date.now() + 60_000;
167 while (Date.now() < deadline) {
168 const url = home.openedUrls()[sinceIndex];
169 if (url) return url;
170 await new Promise((tick) => setTimeout(tick, 250));
171 }
172 throw new Error("inference: opencode never opened an authorization URL");
173};
174
175// ---------------------------------------------------------------------------
176// Run
177// ---------------------------------------------------------------------------
178
179export interface InferenceOptions {
180 /** Provider/model id, e.g. `opencode/deepseek-v4-flash`. */
181 readonly model: string;
182 readonly prompt: string;
183 /** Kill the run after this many ms (default 240s). */
184 readonly timeoutMs?: number;
185 /** Expose one MCP server to the model, with a consent strategy. */
186 readonly mcp?: McpWiring;
187}
188
189const DEFAULT_TIMEOUT_MS = 240_000;
190
191export const runInference = async (options: InferenceOptions): Promise<InferenceResult> => {
192 const home = makeInferenceHome(options.mcp);
193 if (options.mcp) await connectMcp(home, options.mcp);
194 const startedAt = Date.now();
195
196 const child = spawn(
197 "opencode",
198 ["run", "-m", options.model, "--format", "json", options.prompt],
199 {
200 cwd: home.projectDir,
201 // PWD must match cwd: an inherited PWD pointing at this repo invites the
202 // model to wander the codebase instead of acting in an empty project.
203 env: { ...home.env, PWD: home.projectDir },
204 stdio: ["ignore", "pipe", "pipe"],
205 },
206 );
207
208 let stdout = "";
209 let stderr = "";
210 child.stdout.on("data", (chunk: Buffer) => (stdout += chunk.toString("utf8")));
211 child.stderr.on("data", (chunk: Buffer) => (stderr += chunk.toString("utf8")));
212
213 // Keep answering browser hops for the whole run — the model may (re)connect
214 // the MCP server mid-conversation, not just at pre-auth.
215 let consented = home.openedUrls().length;
216 const consentLoop = options.mcp
217 ? setInterval(() => {
218 const urls = home.openedUrls();
219 if (urls.length > consented) {
220 const index = consented;
221 consented = urls.length;
222 void options.mcp?.consent(urls[index]).catch(() => {});
223 }
224 }, 300)
225 : undefined;
226
227 const exitCode = await new Promise<number | null>((resolve) => {
228 const killer = setTimeout(() => child.kill("SIGKILL"), options.timeoutMs ?? DEFAULT_TIMEOUT_MS);
229 child.once("exit", (code) => {
230 clearTimeout(killer);
231 resolve(code);
232 });
233 });
234 if (consentLoop) clearInterval(consentLoop);
235
236 const events = parseEvents(stdout);
237 return {
238 events,
239 answerText: answerTextOf(events),
240 toolNames: toolNamesOf(events),
241 rawStdout: stdout.length > 0 ? stdout : stderr,
242 exitCode,
243 durationMs: Date.now() - startedAt,
244 };
245};

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t