Extract a reusable dev-inference primitive + infer CLI

The eval harness's core capability — drive the real OpenCode binary with the machine's subscription, hermetically, and read structured output — is useful on its own, so pull it out of the eval-specific harness. - e2e/src/clients/inference.ts: runInference({ model, prompt, mcp? }) → { answerText, events, toolNames, ... }. Hermetic OpenCode home with the subscription credential copied in; MCP optional with a pluggable consent strategy. No effect on the developer's own OpenCode state. - e2e/scripts/infer.ts (+ `bun run infer`): the CLI an agent runs while developing — `bun e2e/scripts/infer.ts -m opencode/glm-5.1 "..."`. - evals/harness.ts is now a thin consumer: runTrial = runInference + the selfhost cookie-consent strategy; grading vocabulary unchanged. Also: drop the connect-handoff 'baseUrl resolved' check — running against current main surfaced that integration-level baseUrl is override-only now (per-operation baseUrl from the #968-970 storage refactors), so the check asserted an obsolete field. Documented in EVALS.md. Switch the eval test import to @effect/vitest per the no-vitest-import rule.

Rhys Sullivan committed 5 months ago

Commit 0295eec1 parent c396d0d7 files · +404 −231Browse files

Unified · Split

7 files

e2e/evals/EVALS.md +31 −0
e2e/evals/evals.test.ts +1 −1
e2e/evals/harness.ts +44 −224
e2e/evals/tasks.ts +8 −5
e2e/package.json +2 −1
e2e/scripts/infer.ts +73 −0
e2e/src/clients/inference.ts +245 −0

e2e/evals/EVALS.md+31−0

@@ -6,6 +6,26 @@ ask, does a real agent driving our real MCP server land on the behavior we · 5 unmodified lines
6	6	designed?_ The tool descriptions are the only steering; if a task needs
7	7	system-prompt coaching to pass, that's a finding about the descriptions.
8	8
	9	+## The inference primitive (reusable on its own)
	10	+
	11	+The eval harness is built on a standalone helper that any agent or test working
	12	+in this repo can use to run real inference, hermetically:
	13	+
	14	+```sh
	15	+# Ask a model a question through the OpenCode subscription — no effect on your
	16	+# own OpenCode history:
	17	+bun e2e/scripts/infer.ts "Reply with exactly: pong"
	18	+bun e2e/scripts/infer.ts -m opencode/glm-5.1 "Summarize this stack trace: ..."
	19	+bun e2e/scripts/infer.ts --json "..." # full JSON event stream
	20	+cd e2e && bun run infer "..." # via the package script
	21	+```
	22	+
	23	+Programmatically, `runInference({ model, prompt, mcp? })` from
	24	+`e2e/src/clients/inference.ts` returns `{ answerText, events, toolNames, ... }`.
	25	+Pass `mcp` to expose one of our MCP servers to the model (with a consent
	26	+strategy); omit it for plain question→answer. The eval harness is just this
	27	+primitive plus grading.
	28	+
9	29	## How it works
10	30
11	31	- Client: the real OpenCode binary (`opencode run --format json`), in a
@@ -126,3 +146,14 @@ description tweaks made because of it): · 114 unmodified lines
126	146	output/log scrubbing is the pure-upside piece and the cred-hygiene task can
127	147	A/B any candidate (it already separates "secret in tool call" vs "secret in
128	148	answer"). Tracking this as a known gap, not a bug to fix now.
	149	+- 2026-06-12 · integration-level baseUrl is now override-only: against
	150	+ current main, an add-by-spec-URL stores `config.baseUrl: null` even though
	151	+ the spec declares `servers[0].url`. Not a regression — the storage refactors
	152	+ (#968–970) moved the host to PER-OPERATION baseUrl (baked into each compiled
	153	+ tool from the spec's `servers`), so tool calls still reach the right host and
	154	+ integration-level baseUrl became an override-only field. The connect-handoff
	155	+ task's old "baseUrl resolved to the emulator" check asserted the obsolete
	156	+ integration-level field and started failing the moment the eval ran against
	157	+ main instead of the PR #957 branch — exactly the kind of semantic drift the
	158	+ eval exists to surface. Check removed; "registered + apikey derived" already
	159	+ proves the spec compiled with auth.

e2e/evals/evals.test.ts+1−1

@@ -5,7 +5,7 @@ · 4 unmodified lines
5	5	import { mkdirSync, writeFileSync } from "node:fs";
6	6	import { join } from "node:path";
7	7
8		−import { describe, expect, it } from "vitest";
	8	+import { describe, expect, it } from "@effect/vitest";
9	9
10	10	import { resolveTarget } from "../targets/registry";
11	11	import { hasOpenCode } from "../src/clients/opencode";

e2e/evals/harness.ts+44−224

@@ -1,27 +1,29 @@
1		−// The eval harness: run the REAL OpenCode binary with real Go-subscription
2		−// inference against a real target, then grade the result with deterministic
3		−// checks. One eval = one task × one model × one trial; the vitest file fans
4		−// out the matrix and aggregates pass rates (see report.ts).
	1	+// The eval harness: run a real OpenCode inference trial against a real target,
	2	+// then grade the result with deterministic checks. One eval = one task × one
	3	+// model × one trial; the vitest file fans out the matrix and aggregates pass
	4	+// rates. The inference mechanics live in the reusable `runInference` primitive
	5	+// (src/clients/inference.ts); this file adds only the target-specific MCP
	6	+// consent strategy and the grading vocabulary.
5	7	//
6	8	// Design intent (EVALS.md): the agent gets the user's one-line ask and
7	9	// whatever our MCP server advertises — no extra system prompt, no coached
8	10	// tool order. The tool descriptions are what's under test.
9		−import { spawn, spawnSync } from "node:child_process";
10		−import { copyFileSync, existsSync, mkdirSync, writeFileSync } from "node:fs";
11		−import { homedir } from "node:os";
12		−import { join } from "node:path";
13		−
14	11	import { Effect } from "effect";
15	12
16	13	import type { Identity, Target } from "../src/target";
17		−import { makeOpenCodeHome, warmUp, type OpenCodeHome } from "../src/clients/opencode";
	14	+import {
	15	+ hasOpenCodeSubscription,
	16	+ runInference,
	17	+ toolTrafficOf,
	18	+ type InferenceResult,
	19	+} from "../src/clients/inference";
18	20
19	21	// ---------------------------------------------------------------------------
20	22	// Config — every knob is an env var so CI and local runs share one path.
21	23	// ---------------------------------------------------------------------------
22	24
23	25	export const EVAL_DEFAULT_MODELS = [
24		− // Spread across separate Go quota pools; see EVALS.md for the full table.
	26	+ // Spread across separate subscription quota pools; see EVALS.md for the table.
25	27	"opencode/deepseek-v4-flash",
26	28	"opencode/minimax-m2.5",
27	29	"opencode/kimi-k2.5",
@@ -39,235 +41,53 @@ export const evalTrials = (): number => { · 11 unmodified lines
39	41	return Number.isInteger(parsed) && parsed > 0 ? parsed : 3;
40	42	};
41	43
42		−/** The host machine's Go credential, copied into each hermetic home so the
43		− * throwaway OpenCode can use the subscription. */
44		−const hostAuthFile = (): string => join(homedir(), ".local", "share", "opencode", "auth.json");
45		−
46		−export const hasGoSubscription = (): boolean => existsSync(hostAuthFile());
47		−
48		−// ---------------------------------------------------------------------------
49		−// One trial: spawn `opencode run` headless, collect the JSON event stream.
50		−// ---------------------------------------------------------------------------
51		−
52		−export interface TrialEvent {
53		− readonly type: string;
54		− readonly part?: {
55		− readonly type?: string;
56		− readonly text?: string;
57		− readonly tool?: string;
58		− readonly state?: {
59		− readonly status?: string;
60		− readonly input?: unknown;
61		− readonly output?: unknown;
62		− };
63		− };
64		−}
65		−
66		−export interface TrialResult {
67		− /** Every JSON event opencode emitted, in order. */
68		− readonly events: readonly TrialEvent[];
69		− /** All assistant text parts joined — "what the user read". */
70		− readonly answerText: string;
71		− /** Raw stdout (JSONL) for the artifact dir. */
72		− readonly rawStdout: string;
73		− readonly exitCode: number \| null;
74		− readonly durationMs: number;
75		−}
76		−
77		−export const trialAnswerText = (events: readonly TrialEvent[]): string =>
78		− events
79		− .filter((e) => e.type === "text" && typeof e.part?.text === "string")
80		− .map((e) => e.part?.text ?? "")
81		− .join("\n");
	44	+export const hasGoSubscription = hasOpenCodeSubscription;
82	45
83		−/** Tool-call inputs/outputs as strings, for transcript-wide content checks
84		− * (e.g. "the credential never appears anywhere the model produced"). */
85		−export const trialToolTraffic = (events: readonly TrialEvent[]): string =>
86		− events
87		− .filter((e) => e.type === "tool_use")
88		− .map((e) => JSON.stringify(e.part?.state ?? {}))
89		− .join("\n");
	46	+// Re-exported for grading code (tasks.ts) that inspects tool traffic.
	47	+export const trialToolTraffic = toolTrafficOf;
90	48
91		−/** Names of tools the model invoked, for "used our MCP tools at all" checks. */
92		−export const trialToolNames = (events: readonly TrialEvent[]): readonly string[] =>
93		− events.filter((e) => e.type === "tool_use").map((e) => e.part?.tool ?? "");
	49	+export type TrialResult = InferenceResult;
94	50
95	51	export interface RunTrialOptions {
96	52	readonly serverName: string;
97	53	readonly mcpUrl: string;
98	54	readonly model: string;
99	55	readonly prompt: string;
100		− /** Identity whose email answers the MCP OAuth consent hop. */
	56	+ /** Identity whose session cookie answers the MCP OAuth consent hop. */
101	57	readonly identity: Identity;
102	58	readonly timeoutMs: number;
103	59	}
104	60
105		−/** A hermetic OpenCode home wired for real inference: the target's MCP server
106		− * plus the host's Go credential. Tool permissions are pre-allowed — evals
107		− * measure model behavior, not consent dialogs. */
108		−const makeEvalHome = (serverName: string, mcpUrl: string): OpenCodeHome => {
109		− const home = makeOpenCodeHome(serverName, mcpUrl);
110		− const authDir = join(home.env.XDG_DATA_HOME ?? "", "opencode");
111		− mkdirSync(authDir, { recursive: true });
112		− copyFileSync(hostAuthFile(), join(authDir, "auth.json"));
113		− // Extend the generated opencode.json: keep the MCP server, allow all tools,
114		− // disable share/autoupdate noise.
115		− const configPath = join(home.projectDir, "opencode.json");
116		− writeFileSync(
117		− configPath,
118		− JSON.stringify({
119		− $schema: "https://opencode.ai/config.json",
120		− autoupdate: false,
121		− share: "disabled",
122		− permission: { "*": "allow" },
123		− mcp: { [serverName]: { type: "remote", url: mcpUrl } },
124		− }),
125		− );
126		− return home;
127		−};
128		−
129		−/** Play the signed-in human for OpenCode's recorded browser hop: sign in for
130		− * a Better Auth session cookie, drive the authorize URL with it, and deliver
131		− * the resulting code to OpenCode's localhost callback. (The scenario-side
132		− * completeOAuthConsent uses login_hint — that's the cloud emulator's dialect;
133		− * selfhost's Better Auth consent requires the cookie.) */
134		−const consentWithCookie = async (
135		− home: OpenCodeHome,
136		− identity: Identity,
137		− baseUrl: string,
138		− sinceIndex: number,
139		−): Promise<void> => {
140		− const deadline = Date.now() + 60_000;
141		− while (Date.now() < deadline) {
142		− const authorizationUrl = home.openedUrls()[sinceIndex];
143		− if (authorizationUrl) {
144		− const cookie = identity.headers?.cookie ?? "";
145		− const authorize = await fetch(authorizationUrl, {
146		− headers: { cookie },
147		− redirect: "manual",
148		− });
149		− const location = authorize.headers.get("location");
150		− if (!location) {
151		− throw new Error(`eval consent: authorize did not redirect (${authorize.status})`);
152		− }
153		− // Hand the code to OpenCode's local callback server.
154		− const callback = await fetch(location);
155		− if (!callback.ok) throw new Error(`eval consent: callback failed (${callback.status})`);
156		− return;
	61	+/** Answer OpenCode's recorded browser hop the way a signed-in selfhost user
	62	+ * would: drive the authorize URL with the identity's Better Auth session
	63	+ * cookie and deliver the code to OpenCode's localhost callback. (Cloud's
	64	+ * emulator dialect uses login_hint instead — this is the selfhost path.) */
	65	+const cookieConsent =
	66	+ (identity: Identity) =>
	67	+ async (authorizationUrl: string): Promise<void> => {
	68	+ const cookie = identity.headers?.cookie ?? "";
	69	+ const authorize = await fetch(authorizationUrl, { headers: { cookie }, redirect: "manual" });
	70	+ const location = authorize.headers.get("location");
	71	+ if (!location) {
	72	+ throw new Error(`eval consent: authorize did not redirect (${authorize.status})`);
157	73	}
158		− await new Promise((tick) => setTimeout(tick, 250));
159		− }
160		− throw new Error("eval consent: opencode never opened an authorization URL");
161		−};
162		−
163		−/** Connect OpenCode to the target's MCP server before the trial — a user's
164		− * OpenCode is already authenticated by the time they ask for work, and
165		− * `opencode run` does not initiate MCP OAuth itself (without this, the
166		− * executor tools simply never exist and the model free-styles with bash). */
167		−const preAuthMcp = async (
168		− home: OpenCodeHome,
169		− serverName: string,
170		− identity: Identity,
171		− baseUrl: string,
172		−): Promise<void> => {
173		− // First-run database migration in a bare project — `mcp auth` misbehaves
174		− // if it doubles as first run (see warmUp's doc comment).
175		− warmUp(home);
176		− // The auth command must run ASYNC (spawn, not spawnSync): the consent
177		− // helper polls on timers, and a blocked event loop would starve it while
178		− // `mcp auth` sits waiting for the browser hop it recorded via the shim.
179		− const sinceIndex = home.openedUrls().length;
180		− const auth = spawn("opencode", ["mcp", "auth", serverName], {
181		− cwd: home.projectDir,
182		− env: home.env,
183		− stdio: ["ignore", "pipe", "pipe"],
184		− });
185		− const authExit = new Promise<void>((resolve) => {
186		− const killer = setTimeout(() => auth.kill("SIGKILL"), 90_000);
187		− auth.once("exit", () => {
188		− clearTimeout(killer);
189		− resolve();
190		− });
191		− });
192		− await consentWithCookie(home, identity, baseUrl, sinceIndex);
193		− await authExit;
194		− const listed = spawnSync("opencode", ["mcp", "list"], {
195		− cwd: home.projectDir,
196		− env: home.env,
197		− timeout: 60_000,
198		− encoding: "utf8",
199		− });
200		− if (!`${listed.stdout}`.includes("connected")) {
201		− throw new Error(`eval pre-auth: MCP server never reached "connected" for ${serverName}`);
202		− }
203		−};
	74	+ const callback = await fetch(location);
	75	+ if (!callback.ok) throw new Error(`eval consent: callback failed (${callback.status})`);
	76	+ };
204	77
205	78	export const runTrial = (options: RunTrialOptions): Effect.Effect<TrialResult, Error> =>
206		− Effect.promise(async () => {
207		− const home = makeEvalHome(options.serverName, options.mcpUrl);
208		− const baseUrl = new URL(options.mcpUrl).origin;
209		− await preAuthMcp(home, options.serverName, options.identity, baseUrl);
210		− const startedAt = Date.now();
211		−
212		− const child = spawn(
213		− "opencode",
214		− ["run", "-m", options.model, "--format", "json", options.prompt],
215		− {
216		− cwd: home.projectDir,
217		− // PWD must match cwd: the inherited value points at the eval RUNNER's
218		− // checkout, and a leaked path invites the model to wander our repo
219		− // instead of acting like a user in an empty project.
220		− env: { ...home.env, PWD: home.projectDir },
221		− stdio: ["ignore", "pipe", "pipe"],
	79	+ Effect.promise(() =>
	80	+ runInference({
	81	+ model: options.model,
	82	+ prompt: options.prompt,
	83	+ timeoutMs: options.timeoutMs,
	84	+ mcp: {
	85	+ serverName: options.serverName,
	86	+ url: options.mcpUrl,
	87	+ consent: cookieConsent(options.identity),
222	88	},
223		− );
224		−
225		− let stdout = "";
226		− let stderr = "";
227		− child.stdout.on("data", (chunk: Buffer) => (stdout += chunk.toString("utf8")));
228		− child.stderr.on("data", (chunk: Buffer) => (stderr += chunk.toString("utf8")));
229		−
230		− // Play the signed-in human whenever OpenCode opens an OAuth consent URL.
231		− // Pre-auth already granted the MCP session; this loop stays alive for the
232		− // whole trial in case the agent triggers another browser hop mid-run.
233		− let consented = home.openedUrls().length;
234		− const consentLoop = setInterval(() => {
235		− const urls = home.openedUrls();
236		− if (urls.length > consented) {
237		− const index = consented;
238		− consented = urls.length;
239		− void consentWithCookie(home, options.identity, baseUrl, index).catch(() => {});
240		− }
241		− }, 300);
242		−
243		− const exitCode = await new Promise<number \| null>((resolve) => {
244		− const killer = setTimeout(() => child.kill("SIGKILL"), options.timeoutMs);
245		− child.once("exit", (code) => {
246		− clearTimeout(killer);
247		− resolve(code);
248		− });
249		− });
250		− clearInterval(consentLoop);
251		−
252		− const events: TrialEvent[] = [];
253		− for (const line of stdout.split("\n")) {
254		− if (!line.trim()) continue;
255		− // oxlint-disable-next-line executor/no-try-catch-or-throw, executor/no-json-parse -- boundary: tolerant parse of opencode's JSONL event stream
256		− try {
257		− events.push(JSON.parse(line) as TrialEvent);
258		− } catch {
259		− // Non-JSON line (banner, warning) — keep going.
260		− }
261		− }
262		−
263		− return {
264		− events,
265		− answerText: trialAnswerText(events),
266		− rawStdout: stdout.length > 0 ? stdout : stderr,
267		− exitCode,
268		− durationMs: Date.now() - startedAt,
269		− };
270		− });
	89	+ }),
	90	+ );
271	91
272	92	// ---------------------------------------------------------------------------
273	93	// Task registry — a task is a prompt plus deterministic graders.

e2e/evals/tasks.ts+8−5

@@ -50,10 +50,18 @@ const integrationChecks = async ( · 49 unmodified lines
50	50	ctx: GradeContext,
51	51	integration: string,
52	52	): Promise<readonly GradeCheck[]> => {
	53	+ // The config endpoint returns non-null only once addSpec's transaction
	54	+ // committed (integration row + operations together), so this doubles as the
	55	+ // "operations compiled" signal.
53	56	const config = (await ctx
54	57	.apiGet(`/api/openapi/integrations/${integration}/config`)
55	58	.catch(() => null)) as OpenApiConfig \| null;
56	59	const methods = config?.authenticationTemplate ?? [];
	60	+ // NOTE: we intentionally do NOT assert the integration-level `config.baseUrl`.
	61	+ // The host moved to per-operation baseUrl (baked into each tool at compile
	62	+ // time from the spec's `servers`); integration-level baseUrl is now an
	63	+ // override-only field and is null for a plain add-by-spec. The eval surfaced
	64	+ // this when it started running against current main — see EVALS.md.
57	65	return [
58	66	check("integration registered under the asked-for slug", config !== null),
59	67	check(
@@ -61,11 +69,6 @@ const integrationChecks = async ( · 1 unmodified line
61	69	methods.some((m) => m.kind === "apikey"),
62	70	`methods: ${JSON.stringify(methods.map((m) => m.kind))}`,
63	71	),
64		− check(
65		− "baseUrl resolved to the emulator",
66		− (config?.baseUrl ?? "").startsWith(EMULATOR_BASE),
67		− `baseUrl: ${config?.baseUrl}`,
68		− ),
69	72	];
70	73	};
71	74

e2e/package.json+2−1

@@ -12,7 +12,8 @@ · 11 unmodified lines
12	12	"serve": "bun scripts/rebuild-viewer.ts && bun scripts/serve.ts",
13	13	"typecheck": "tsc --noEmit",
14	14	"test:desktop": "vitest run --project desktop",
15		− "test:evals": "EVAL=1 vitest run --project evals"
	15	+ "test:evals": "EVAL=1 vitest run --project evals",
	16	+ "infer": "bun scripts/infer.ts"
16	17	},
17	18	"dependencies": {
18	19	"@executor-js/api": "workspace:*",

e2e/scripts/infer.ts+73−0

@@ -0,0 +1,73 @@
	1	+// Dev-time inference for agents working in this repo: ask a real model a
	2	+// question through the machine's OpenCode subscription, hermetically (no effect
	3	+// on your own OpenCode history). Prints the model's answer to stdout.
	4	+//
	5	+// bun e2e/scripts/infer.ts "What is 6 * 7?"
	6	+// bun e2e/scripts/infer.ts -m opencode/glm-5.1 "Summarize this error: ..."
	7	+// bun e2e/scripts/infer.ts --json "..." # full JSON event stream
	8	+// cd e2e && bun run infer "..." # via the package script
	9	+//
	10	+// Model ids are `opencode/<name>` — see e2e/evals/EVALS.md for the
	11	+// subscription model/quota table. Default is a cheap, high-quota model.
	12	+import { hasOpenCodeSubscription, runInference } from "../src/clients/inference";
	13	+import { hasOpenCode } from "../src/clients/opencode";
	14	+
	15	+const DEFAULT_MODEL = "opencode/deepseek-v4-flash";
	16	+
	17	+const parseArgs = (argv: readonly string[]) => {
	18	+ let model = DEFAULT_MODEL;
	19	+ let json = false;
	20	+ const rest: string[] = [];
	21	+ for (let i = 0; i < argv.length; i++) {
	22	+ const arg = argv[i];
	23	+ if (arg === "-m" \|\| arg === "--model") {
	24	+ model = argv[++i] ?? model;
	25	+ } else if (arg === "--json") {
	26	+ json = true;
	27	+ } else {
	28	+ rest.push(arg);
	29	+ }
	30	+ }
	31	+ return { model, json, prompt: rest.join(" ").trim() };
	32	+};
	33	+
	34	+const main = async () => {
	35	+ if (!hasOpenCode()) {
	36	+ process.stderr.write("infer: the `opencode` binary is not installed.\n");
	37	+ process.exit(2);
	38	+ }
	39	+ if (!hasOpenCodeSubscription()) {
	40	+ process.stderr.write(
	41	+ "infer: no OpenCode credential found (~/.local/share/opencode/auth.json). Run `opencode auth login` first.\n",
	42	+ );
	43	+ process.exit(2);
	44	+ }
	45	+
	46	+ const { model, json, prompt } = parseArgs(process.argv.slice(2));
	47	+ if (!prompt) {
	48	+ process.stderr.write(
	49	+ 'infer: no prompt. Usage: bun e2e/scripts/infer.ts [-m model] "your prompt"\n',
	50	+ );
	51	+ process.exit(2);
	52	+ }
	53	+
	54	+ const result = await runInference({ model, prompt });
	55	+ if (json) {
	56	+ process.stdout.write(
	57	+ result.rawStdout.endsWith("\n") ? result.rawStdout : `${result.rawStdout}\n`,
	58	+ );
	59	+ } else {
	60	+ process.stdout.write(`${result.answerText}\n`);
	61	+ }
	62	+ if (result.exitCode !== 0) {
	63	+ process.stderr.write(
	64	+ `infer: opencode exited ${result.exitCode} after ${result.durationMs}ms\n`,
	65	+ );
	66	+ }
	67	+};
	68	+
	69	+// oxlint-disable-next-line executor/no-try-catch-or-throw -- boundary: top-level CLI error handler
	70	+main().catch((error) => {
	71	+ process.stderr.write(`infer: ${String(error)}\n`);
	72	+ process.exit(1);
	73	+});

e2e/src/clients/inference.ts+245−0

@@ -0,0 +1,245 @@
	1	+// Dev-time inference: drive the REAL OpenCode binary as a one-shot model call,
	2	+// hermetically. A throwaway OpenCode home borrows the machine's OpenCode
	3	+// subscription credential, so an agent (or a test) working in this repo can
	4	+// ask a real model a question — optionally with one of our MCP servers wired
	5	+// in — and get structured output back, without touching the developer's own
	6	+// OpenCode state/history.
	7	+//
	8	+// Two consumers:
	9	+// - `e2e/scripts/infer.ts` — the CLI an agent runs while developing.
	10	+// - `e2e/evals/` — the eval harness, which adds grading on top.
	11	+//
	12	+// What OpenCode does with the prompt, the tools, and (when wired) MCP OAuth is
	13	+// entirely its own code; we only provide the credential and read the result.
	14	+import { spawn, spawnSync } from "node:child_process";
	15	+import { copyFileSync, existsSync, mkdirSync, writeFileSync } from "node:fs";
	16	+import { homedir } from "node:os";
	17	+import { join } from "node:path";
	18	+
	19	+import { makeOpenCodeHome, warmUp, type OpenCodeHome } from "./opencode";
	20	+
	21	+// ---------------------------------------------------------------------------
	22	+// Subscription credential
	23	+// ---------------------------------------------------------------------------
	24	+
	25	+/** The machine's OpenCode credential (e.g. the Go subscription), copied into
	26	+ * each hermetic home so the throwaway OpenCode can run inference. */
	27	+const hostAuthFile = (): string => join(homedir(), ".local", "share", "opencode", "auth.json");
	28	+
	29	+export const hasOpenCodeSubscription = (): boolean => existsSync(hostAuthFile());
	30	+
	31	+// ---------------------------------------------------------------------------
	32	+// Event stream
	33	+// ---------------------------------------------------------------------------
	34	+
	35	+export interface InferenceEvent {
	36	+ readonly type: string;
	37	+ readonly part?: {
	38	+ readonly type?: string;
	39	+ readonly text?: string;
	40	+ readonly tool?: string;
	41	+ readonly state?: {
	42	+ readonly status?: string;
	43	+ readonly input?: unknown;
	44	+ readonly output?: unknown;
	45	+ };
	46	+ };
	47	+}
	48	+
	49	+export interface InferenceResult {
	50	+ /** Every JSON event opencode emitted, in order. */
	51	+ readonly events: readonly InferenceEvent[];
	52	+ /** All assistant text parts joined — "what the user read". */
	53	+ readonly answerText: string;
	54	+ /** Names of the tools the model invoked. */
	55	+ readonly toolNames: readonly string[];
	56	+ /** Raw stdout (JSONL), or stderr if stdout was empty — for artifacts. */
	57	+ readonly rawStdout: string;
	58	+ readonly exitCode: number \| null;
	59	+ readonly durationMs: number;
	60	+}
	61	+
	62	+export const answerTextOf = (events: readonly InferenceEvent[]): string =>
	63	+ events
	64	+ .filter((e) => e.type === "text" && typeof e.part?.text === "string")
	65	+ .map((e) => e.part?.text ?? "")
	66	+ .join("\n");
	67	+
	68	+/** Tool-call inputs/outputs as one string, for content checks (e.g. "the
	69	+ * credential never appears anywhere the model produced"). */
	70	+export const toolTrafficOf = (events: readonly InferenceEvent[]): string =>
	71	+ events
	72	+ .filter((e) => e.type === "tool_use")
	73	+ .map((e) => JSON.stringify(e.part?.state ?? {}))
	74	+ .join("\n");
	75	+
	76	+export const toolNamesOf = (events: readonly InferenceEvent[]): readonly string[] =>
	77	+ events.filter((e) => e.type === "tool_use").map((e) => e.part?.tool ?? "");
	78	+
	79	+const parseEvents = (stdout: string): InferenceEvent[] => {
	80	+ const events: InferenceEvent[] = [];
	81	+ for (const line of stdout.split("\n")) {
	82	+ if (!line.trim()) continue;
	83	+ // oxlint-disable-next-line executor/no-try-catch-or-throw, executor/no-json-parse -- boundary: tolerant parse of opencode's JSONL event stream
	84	+ try {
	85	+ events.push(JSON.parse(line) as InferenceEvent);
	86	+ } catch {
	87	+ // Non-JSON line (banner, warning) — keep going.
	88	+ }
	89	+ }
	90	+ return events;
	91	+};
	92	+
	93	+// ---------------------------------------------------------------------------
	94	+// Hermetic home
	95	+// ---------------------------------------------------------------------------
	96	+
	97	+/** Optional MCP server to expose to the model. `consent` answers OpenCode's
	98	+ * recorded browser hop: given the authorization URL it opened, deliver the
	99	+ * code to OpenCode's local callback (the strategy is target-specific —
	100	+ * Better Auth cookie, a login_hint redirect, etc.). */
	101	+export interface McpWiring {
	102	+ readonly serverName: string;
	103	+ readonly url: string;
	104	+ readonly consent: (authorizationUrl: string) => Promise<void>;
	105	+}
	106	+
	107	+/** A throwaway OpenCode home with the subscription credential and all tool
	108	+ * permissions pre-allowed (inference measures model behavior, not consent
	109	+ * dialogs). Wires in one MCP server when `mcp` is provided. */
	110	+const makeInferenceHome = (mcp?: McpWiring): OpenCodeHome => {
	111	+ const home = makeOpenCodeHome(mcp?.serverName ?? "none", mcp?.url ?? "http://127.0.0.1:0/mcp");
	112	+ const authDir = join(home.env.XDG_DATA_HOME ?? "", "opencode");
	113	+ mkdirSync(authDir, { recursive: true });
	114	+ copyFileSync(hostAuthFile(), join(authDir, "auth.json"));
	115	+ writeFileSync(
	116	+ join(home.projectDir, "opencode.json"),
	117	+ JSON.stringify({
	118	+ $schema: "https://opencode.ai/config.json",
	119	+ autoupdate: false,
	120	+ share: "disabled",
	121	+ permission: { "*": "allow" },
	122	+ ...(mcp ? { mcp: { [mcp.serverName]: { type: "remote", url: mcp.url } } } : {}),
	123	+ }),
	124	+ );
	125	+ return home;
	126	+};
	127	+
	128	+/** Connect OpenCode to an MCP server before the run — `opencode run` does not
	129	+ * initiate MCP OAuth itself, so without this the server's tools never exist
	130	+ * and the model free-styles with its built-ins. Drives `opencode mcp auth`
	131	+ * and answers the browser hop via `mcp.consent`. */
	132	+const connectMcp = async (home: OpenCodeHome, mcp: McpWiring): Promise<void> => {
	133	+ // First-run DB migration in a bare project — `mcp auth` misbehaves if it
	134	+ // doubles as first run (see warmUp's doc comment).
	135	+ warmUp(home);
	136	+ // ASYNC spawn (not spawnSync): the consent step polls on timers, and a
	137	+ // blocked event loop would starve it while `mcp auth` waits for the browser
	138	+ // hop it records via the open(1) shim.
	139	+ const sinceIndex = home.openedUrls().length;
	140	+ const auth = spawn("opencode", ["mcp", "auth", mcp.serverName], {
	141	+ cwd: home.projectDir,
	142	+ env: home.env,
	143	+ stdio: ["ignore", "pipe", "pipe"],
	144	+ });
	145	+ const authExit = new Promise<void>((resolve) => {
	146	+ const killer = setTimeout(() => auth.kill("SIGKILL"), 90_000);
	147	+ auth.once("exit", () => {
	148	+ clearTimeout(killer);
	149	+ resolve();
	150	+ });
	151	+ });
	152	+ await waitForAuthorizationUrl(home, sinceIndex).then(mcp.consent);
	153	+ await authExit;
	154	+ const listed = spawnSync("opencode", ["mcp", "list"], {
	155	+ cwd: home.projectDir,
	156	+ env: home.env,
	157	+ timeout: 60_000,
	158	+ encoding: "utf8",
	159	+ });
	160	+ if (!`${listed.stdout}`.includes("connected")) {
	161	+ throw new Error(`inference: MCP server "${mcp.serverName}" never reached "connected"`);
	162	+ }
	163	+};
	164	+
	165	+const waitForAuthorizationUrl = async (home: OpenCodeHome, sinceIndex: number): Promise<string> => {
	166	+ const deadline = Date.now() + 60_000;
	167	+ while (Date.now() < deadline) {
	168	+ const url = home.openedUrls()[sinceIndex];
	169	+ if (url) return url;
	170	+ await new Promise((tick) => setTimeout(tick, 250));
	171	+ }
	172	+ throw new Error("inference: opencode never opened an authorization URL");
	173	+};
	174	+
	175	+// ---------------------------------------------------------------------------
	176	+// Run
	177	+// ---------------------------------------------------------------------------
	178	+
	179	+export interface InferenceOptions {
	180	+ /** Provider/model id, e.g. `opencode/deepseek-v4-flash`. */
	181	+ readonly model: string;
	182	+ readonly prompt: string;
	183	+ /** Kill the run after this many ms (default 240s). */
	184	+ readonly timeoutMs?: number;
	185	+ /** Expose one MCP server to the model, with a consent strategy. */
	186	+ readonly mcp?: McpWiring;
	187	+}
	188	+
	189	+const DEFAULT_TIMEOUT_MS = 240_000;
	190	+
	191	+export const runInference = async (options: InferenceOptions): Promise<InferenceResult> => {
	192	+ const home = makeInferenceHome(options.mcp);
	193	+ if (options.mcp) await connectMcp(home, options.mcp);
	194	+ const startedAt = Date.now();
	195	+
	196	+ const child = spawn(
	197	+ "opencode",
	198	+ ["run", "-m", options.model, "--format", "json", options.prompt],
	199	+ {
	200	+ cwd: home.projectDir,
	201	+ // PWD must match cwd: an inherited PWD pointing at this repo invites the
	202	+ // model to wander the codebase instead of acting in an empty project.
	203	+ env: { ...home.env, PWD: home.projectDir },
	204	+ stdio: ["ignore", "pipe", "pipe"],
	205	+ },
	206	+ );
	207	+
	208	+ let stdout = "";
	209	+ let stderr = "";
	210	+ child.stdout.on("data", (chunk: Buffer) => (stdout += chunk.toString("utf8")));
	211	+ child.stderr.on("data", (chunk: Buffer) => (stderr += chunk.toString("utf8")));
	212	+
	213	+ // Keep answering browser hops for the whole run — the model may (re)connect
	214	+ // the MCP server mid-conversation, not just at pre-auth.
	215	+ let consented = home.openedUrls().length;
	216	+ const consentLoop = options.mcp
	217	+ ? setInterval(() => {
	218	+ const urls = home.openedUrls();
	219	+ if (urls.length > consented) {
	220	+ const index = consented;
	221	+ consented = urls.length;
	222	+ void options.mcp?.consent(urls[index]).catch(() => {});
	223	+ }
	224	+ }, 300)
	225	+ : undefined;
	226	+
	227	+ const exitCode = await new Promise<number \| null>((resolve) => {
	228	+ const killer = setTimeout(() => child.kill("SIGKILL"), options.timeoutMs ?? DEFAULT_TIMEOUT_MS);
	229	+ child.once("exit", (code) => {
	230	+ clearTimeout(killer);
	231	+ resolve(code);
	232	+ });
	233	+ });
	234	+ if (consentLoop) clearInterval(consentLoop);
	235	+
	236	+ const events = parseEvents(stdout);
	237	+ return {
	238	+ events,
	239	+ answerText: answerTextOf(events),
	240	+ toolNames: toolNamesOf(events),
	241	+ rawStdout: stdout.length > 0 ? stdout : stderr,
	242	+ exitCode,
	243	+ durationMs: Date.now() - startedAt,
	244	+ };
	245	+};

Extract a reusable dev-inference primitive + infer CLI

Keyboard shortcuts

General

Issues and pull requests

Issues, pull requests and Inbox

Repository code