The inference command isn't an abstraction, it's a devtool — so it's named like one. `bun run cli pi` and `bun run cli opencode` are the same mental shape as the dev server or the browser: pick the tool, drive it. pi (clients/pi.ts, was agent.ts): runPi() gains resumable sessions — -s NAME maps to a stable session UUID + pinned project dir under e2e/.dev/pi-sessions/<name> (pi sessions are project-scoped, so the cwd must persist with them). Verified: seed a codeword in one process, recall it from another. opencode (clients/opencode.ts): the existing MCP-actor home machinery gains a headless one-shot/multi-turn runner. runOpenCode() drives the real binary hermetically (XDG home seeded with the machine credential), -s NAME pins a reusable home under e2e/.dev/opencode-homes/<name>, --continue resumes its last conversation, and the mcp option wires OpenCode's native MCP client at a target. makeOpenCodeHome learns an optional persistent root (existing config left alone on reattach). Verified: pong, codeword seed + --continue recall, both live. Division of labor stands: pi = everyday inference workhorse (~2s, no MCP by design), opencode = the MCP-native client-behavior actor.
Unified · Split
| @@ -76,7 +76,8 @@ bun run cli identity selfhost # fresh identity (headers / cookies / creds) · 75 unmodified lines | |||
| 76 | bun run cli api selfhost tools.list | 76 | bun run cli api selfhost tools.list |
| 77 | bun run cli mcp selfhost call execute '{"code":"return 1+1;"}' | 77 | bun run cli mcp selfhost call execute '{"code":"return 1+1;"}' |
| 78 | bun run cli ledger cloud workos # what hit the emulator | 78 | bun run cli ledger cloud workos # what hit the emulator |
| 79 | bun run cli infer "..." # real model inference (see below) | 79 | bun run cli pi "..." # real model inference via pi (see below) |
| 80 | bun run cli opencode "..." # the same through the OpenCode binary | ||
| 80 | bun run cli down selfhost # tear down (also removes tailscale serves) | 81 | bun run cli down selfhost # tear down (also removes tailscale serves) |
| 81 | ``` | 82 | ``` |
| 82 | 83 | ||
| @@ -97,20 +98,33 @@ hostname (`__VITE_ADDITIONAL_SERVER_ALLOWED_HOSTS`). · 14 unmodified lines | |||
| 97 | 98 | ||
| 98 | ## Real inference for development | 99 | ## Real inference for development |
| 99 | 100 | ||
| 100 | `bun run cli infer [-m model] [--tools] [--json] "prompt"` runs a real model | 101 | The agent-tool commands are named after the actual tools — they're devtools, |
| 101 | through the **pi agent harness** (`@earendil-works/pi-coding-agent`, | 102 | same as the dev server or the browser. Both ride the machine's OpenCode |
| 102 | headless `--mode json`), reached via the machine's OpenCode Zen subscription | 103 | subscription (`~/.local/share/opencode/auth.json`; run `opencode auth login` |
| 103 | (key auto-discovered from `~/.local/share/opencode/auth.json`; run | 104 | once if missing); both are hermetic (throwaway homes, nothing touches your |
| 104 | `opencode auth login` once if missing). Every run is hermetic: throwaway pi | 105 | own pi/OpenCode state); both take `--json` for the distilled result (answer, |
| 105 | config dir, no session persisted, and `--tools` confines the model's | 106 | events, tool calls) for programmatic/grading use. |
| 106 | read/bash/edit/write to a fresh scratch dir. Default model is cheap+fast | 107 | |
| 107 | (deepseek-v4-flash); `-m glm-5.1` etc. selects others on the subscription. | 108 | `bun run cli pi [-m model] [-s session] [--tools] [--json] "prompt"` — the |
| 108 | `--json` prints the distilled result (answer, thinking, tool calls, usage, | 109 | **pi agent harness** (`@earendil-works/pi-coding-agent`, headless |
| 109 | full event stream) for programmatic use; the same primitive is importable as | 110 | `--mode json`), reaching the Zen gateway through a generated provider |
| 110 | `runAgent()` from `e2e/src/clients/agent.ts` for eval-style harnesses. | 111 | extension. Fast (~2s), `--no-tools` by default; `--tools` confines pi's |
| 111 | OpenCode (`e2e/src/clients/opencode.ts`) remains the MCP-native real-client | 112 | read/bash/edit/write to a scratch dir. `-s NAME` makes the conversation |
| 112 | actor for scenarios that test OAuth/tool-discovery behavior; pi is the | 113 | resumable (state under `e2e/.dev/pi-sessions/<name>`). Programmatic: |
| 113 | inference workhorse. | 114 | `runPi()` in `e2e/src/clients/pi.ts`. pi has no MCP by design — for |
| 115 | MCP-tool work, either drive the typed API/MCP through this CLI's other | ||
| 116 | commands, or use opencode below. | ||
| 117 | |||
| 118 | `bun run cli opencode [-m provider/model] [-s session] [--continue] | ||
| 119 | [--json] "prompt"` — the same ask through the **real OpenCode binary** | ||
| 120 | (hermetic XDG home seeded with the credential). `-s NAME` pins a reusable | ||
| 121 | home under `e2e/.dev/opencode-homes/<name>`; `--continue` resumes its last | ||
| 122 | conversation. OpenCode is also the **MCP-native client**: programmatically, | ||
| 123 | `runOpenCode({ mcp: { serverName, url } })` configures a remote MCP server | ||
| 124 | (OAuth consent is the caller's job — see the `mcp-opencode-real` scenario | ||
| 125 | for the full dance). This is the actor to use when the thing under test is | ||
| 126 | client behavior (tool discovery, OAuth hops); pi is the everyday inference | ||
| 127 | workhorse. | ||
| 114 | 128 | ||
| 115 | ## Environment gotchas (learned the hard way) | 129 | ## Environment gotchas (learned the hard way) |
| 116 | 130 | ||
| @@ -496,33 +496,71 @@ const logs = (targetName: string) => { · 495 unmodified lines | |||
| 496 | console.log(readFileSync(state.logFile, "utf8")); | 496 | console.log(readFileSync(state.logFile, "utf8")); |
| 497 | }; | 497 | }; |
| 498 | 498 | ||
| 499 | // Real inference through the pi agent harness (Zen subscription). Plain | 499 | // Shared arg parsing for the agent-tool commands: -m model, -s session name, |
| 500 | // question→answer by default; --tools gives the model pi's coding tools in a | 500 | // everything else is the prompt. |
| 501 | // scratch dir; --json dumps the full distilled result for grading. | 501 | const parseAgentArgs = (args: ReadonlyArray<string>) => { |
| 502 | const infer = async (args: ReadonlyArray<string>, flags: ReadonlySet<string>) => { | 502 | let model: string | undefined; |
| 503 | const { runAgent, DEFAULT_MODEL, hasInferenceCredential } = await import("../src/clients/agent"); | 503 | let session: string | undefined; |
| 504 | if (!hasInferenceCredential()) { | ||
| 505 | throw new Error( | ||
| 506 | "infer: no inference credential — run `opencode auth login` once on this machine", | ||
| 507 | ); | ||
| 508 | } | ||
| 509 | let model = DEFAULT_MODEL; | ||
| 510 | const prompt: string[] = []; | 504 | const prompt: string[] = []; |
| 511 | for (let index = 0; index < args.length; index++) { | 505 | for (let index = 0; index < args.length; index++) { |
| 512 | if (args[index] === "-m") model = args[++index] ?? model; | 506 | if (args[index] === "-m") model = args[++index]; |
| 507 | else if (args[index] === "-s") session = args[++index]; | ||
| 513 | else prompt.push(args[index]!); | 508 | else prompt.push(args[index]!); |
| 514 | } | 509 | } |
| 515 | if (prompt.length === 0) throw new Error('usage: infer [-m model] [--tools] [--json] "prompt"'); | 510 | return { model, session, prompt: prompt.join(" ") }; |
| 516 | const result = await runAgent({ | 511 | }; |
| 517 | prompt: prompt.join(" "), | 512 | |
| 513 | // `bun run cli pi` — real inference through the pi agent harness (Zen | ||
| 514 | // subscription). Plain question→answer by default; --tools gives the model | ||
| 515 | // pi's coding tools in a scratch dir; -s NAME makes it multi-turn. | ||
| 516 | const piCmd = async (args: ReadonlyArray<string>, flags: ReadonlySet<string>) => { | ||
| 517 | const { runPi, hasInferenceCredential } = await import("../src/clients/pi"); | ||
| 518 | if (!hasInferenceCredential()) { | ||
| 519 | throw new Error("pi: no inference credential — run `opencode auth login` once on this machine"); | ||
| 520 | } | ||
| 521 | const { model, session, prompt } = parseAgentArgs(args); | ||
| 522 | if (!prompt) throw new Error('usage: pi [-m model] [-s session] [--tools] [--json] "prompt"'); | ||
| 523 | const result = await runPi({ prompt, model, session, tools: flags.has("--tools") }); | ||
| 524 | if (flags.has("--json")) { | ||
| 525 | console.log(JSON.stringify(result, null, 2)); | ||
| 526 | } else { | ||
| 527 | if (result.exitCode !== 0 && !result.answerText) { | ||
| 528 | throw new Error(`pi exited ${result.exitCode}\n${result.stderr.slice(0, 800)}`); | ||
| 529 | } | ||
| 530 | console.log(result.answerText); | ||
| 531 | } | ||
| 532 | }; | ||
| 533 | |||
| 534 | // `bun run cli opencode` — the same ask through the real OpenCode binary | ||
| 535 | // (hermetic home, subscription credential seeded). --continue resumes the | ||
| 536 | // hermetic home's last session — pass -s NAME to pick which home. | ||
| 537 | const opencodeCmd = async (args: ReadonlyArray<string>, flags: ReadonlySet<string>) => { | ||
| 538 | const { runOpenCode, hasOpenCode, makeOpenCodeHome } = await import("../src/clients/opencode"); | ||
| 539 | if (!hasOpenCode()) throw new Error("opencode: binary not installed"); | ||
| 540 | const { model, session, prompt } = parseAgentArgs(args); | ||
| 541 | if (!prompt) { | ||
| 542 | throw new Error( | ||
| 543 | 'usage: opencode [-m provider/model] [-s session] [--continue] [--json] "prompt"', | ||
| 544 | ); | ||
| 545 | } | ||
| 546 | // A named "session" here = a reusable hermetic home under .dev, so a later | ||
| 547 | // invocation with the same -s NAME --continue finds the same conversation. | ||
| 548 | const home = session | ||
| 549 | ? makeOpenCodeHome("none", "http://127.0.0.1:9/", { | ||
| 550 | root: join(devDir, "opencode-homes", session), | ||
| 551 | }) | ||
| 552 | : undefined; | ||
| 553 | const result = await runOpenCode({ | ||
| 554 | prompt, | ||
| 518 | model, | 555 | model, |
| 519 | tools: flags.has("--tools"), | 556 | home, |
| 557 | continueSession: flags.has("--continue"), | ||
| 520 | }); | 558 | }); |
| 521 | if (flags.has("--json")) { | 559 | if (flags.has("--json")) { |
| 522 | console.log(JSON.stringify(result, null, 2)); | 560 | console.log(JSON.stringify({ ...result, home: undefined }, null, 2)); |
| 523 | } else { | 561 | } else { |
| 524 | if (result.exitCode !== 0 && !result.answerText) { | 562 | if (result.exitCode !== 0 && !result.answerText) { |
| 525 | throw new Error(`infer: pi exited ${result.exitCode}\n${result.stderr.slice(0, 800)}`); | 563 | throw new Error(`opencode exited ${result.exitCode}\n${result.stderr.slice(0, 800)}`); |
| 526 | } | 564 | } |
| 527 | console.log(result.answerText); | 565 | console.log(result.answerText); |
| 528 | } | 566 | } |
| @@ -540,9 +578,14 @@ const HELP = `e2e dev CLI — the scenario primitives, interactive (see e2e/AGEN · 11 unmodified lines | |||
| 540 | api <target> <group.endpoint> [json] typed API call as a fresh identity | 578 | api <target> <group.endpoint> [json] typed API call as a fresh identity |
| 541 | mcp <target> tools | call <tool> [json] MCP session call | 579 | mcp <target> tools | call <tool> [json] MCP session call |
| 542 | ledger <target> [workos|autumn] the emulator's request ledger (cloud) | 580 | ledger <target> [workos|autumn] the emulator's request ledger (cloud) |
| 543 | infer [-m model] [--tools] [--json] "prompt" real inference via the pi | 581 | pi [-m model] [-s session] [--tools] [--json] "prompt" |
| 544 | agent harness (Zen subscription; --tools gives | 582 | real inference via the pi agent harness (Zen |
| 545 | the model coding tools in a scratch dir) | 583 | subscription); --tools = coding tools in a |
| 584 | scratch dir; -s NAME = resumable conversation | ||
| 585 | opencode [-m provider/model] [-s session] [--continue] [--json] "prompt" | ||
| 586 | the same ask through the real OpenCode binary | ||
| 587 | (hermetic home; native MCP client available | ||
| 588 | programmatically via runOpenCode({mcp})) | ||
| 546 | logs <target> dump the instance's dev-server log | 589 | logs <target> dump the instance's dev-server log |
| 547 | down <target> tear down (kills servers, removes tailscale serves) | 590 | down <target> tear down (kills servers, removes tailscale serves) |
| 548 | 591 | ||
| @@ -569,8 +612,10 @@ const main = async () => { · 20 unmodified lines | |||
| 569 | return mcpCall(args[0] ?? "", args[1], args.slice(2)); | 612 | return mcpCall(args[0] ?? "", args[1], args.slice(2)); |
| 570 | case "ledger": | 613 | case "ledger": |
| 571 | return ledger(args[0] ?? "", args[1]); | 614 | return ledger(args[0] ?? "", args[1]); |
| 572 | case "infer": | 615 | case "pi": |
| 573 | return infer(args, flags); | 616 | return piCmd(args, flags); |
| 617 | case "opencode": | ||
| 618 | return opencodeCmd(args, flags); | ||
| 574 | case "logs": | 619 | case "logs": |
| 575 | return logs(args[0] ?? ""); | 620 | return logs(args[0] ?? ""); |
| 576 | case "down": | 621 | case "down": |
| @@ -3,9 +3,16 @@ · 2 unmodified lines | |||
| 3 | // /mcp, and an `open`(1) shim on PATH so the OAuth browser hop becomes a file | 3 | // /mcp, and an `open`(1) shim on PATH so the OAuth browser hop becomes a file |
| 4 | // we can read instead of a window. What OpenCode does with discovery, scopes, | 4 | // we can read instead of a window. What OpenCode does with discovery, scopes, |
| 5 | // tokens, and refresh is entirely its own code — that is the point. | 5 | // tokens, and refresh is entirely its own code — that is the point. |
| 6 | import { spawnSync } from "node:child_process"; | 6 | import { spawn, spawnSync } from "node:child_process"; |
| 7 | import { existsSync, mkdirSync, mkdtempSync, readFileSync, writeFileSync } from "node:fs"; | 7 | import { |
| 8 | import { tmpdir } from "node:os"; | 8 | copyFileSync, |
| 9 | existsSync, | ||
| 10 | mkdirSync, | ||
| 11 | mkdtempSync, | ||
| 12 | readFileSync, | ||
| 13 | writeFileSync, | ||
| 14 | } from "node:fs"; | ||
| 15 | import { homedir, tmpdir } from "node:os"; | ||
| 9 | import { join } from "node:path"; | 16 | import { join } from "node:path"; |
| 10 | 17 | ||
| 11 | /** Whether the real OpenCode binary is installed — the "opencode" capability. */ | 18 | /** Whether the real OpenCode binary is installed — the "opencode" capability. */ |
| @@ -30,42 +37,48 @@ export interface OpenCodeHome { · 18 unmodified lines | |||
| 30 | * pointing OpenCode's LLM traffic at a local replay brain | 37 | * pointing OpenCode's LLM traffic at a local replay brain |
| 31 | * (clients/replay-brain.ts) and selects it as the model — real agent, | 38 | * (clients/replay-brain.ts) and selects it as the model — real agent, |
| 32 | * scripted conversation. Tool permissions are pre-allowed so the recorded | 39 | * scripted conversation. Tool permissions are pre-allowed so the recorded |
| 33 | * TUI session flows without approval dialogs. */ | 40 | * TUI session flows without approval dialogs. |
| 41 | * | ||
| 42 | * Pass `root` to make the home persistent/reattachable (multi-turn CLI use) | ||
| 43 | * instead of a fresh temp dir; an existing root's config is left alone. */ | ||
| 34 | export const makeOpenCodeHome = ( | 44 | export const makeOpenCodeHome = ( |
| 35 | serverName: string, | 45 | serverName: string, |
| 36 | mcpUrl: string, | 46 | mcpUrl: string, |
| 37 | options?: { readonly chatBrainUrl?: string }, | 47 | options?: { readonly chatBrainUrl?: string; readonly root?: string }, |
| 38 | ): OpenCodeHome => { | 48 | ): OpenCodeHome => { |
| 39 | const root = mkdtempSync(join(tmpdir(), "e2e-opencode-")); | 49 | const root = options?.root ?? mkdtempSync(join(tmpdir(), "e2e-opencode-")); |
| 40 | const projectDir = join(root, "project"); | 50 | const projectDir = join(root, "project"); |
| 41 | const dataDir = join(root, "data"); | 51 | const dataDir = join(root, "data"); |
| 42 | const binDir = join(root, "bin"); | 52 | const binDir = join(root, "bin"); |
| 43 | const openedUrlsFile = join(root, "opened-urls.txt"); | 53 | const openedUrlsFile = join(root, "opened-urls.txt"); |
| 54 | const fresh = !existsSync(projectDir); | ||
| 44 | for (const dir of [projectDir, dataDir, binDir]) mkdirSync(dir, { recursive: true }); | 55 | for (const dir of [projectDir, dataDir, binDir]) mkdirSync(dir, { recursive: true }); |
| 45 | 56 | ||
| 46 | writeFileSync( | 57 | if (fresh) { |
| 47 | join(projectDir, "opencode.json"), | 58 | writeFileSync( |
| 48 | JSON.stringify({ | 59 | join(projectDir, "opencode.json"), |
| 49 | $schema: "https://opencode.ai/config.json", | 60 | JSON.stringify({ |
| 50 | mcp: { [serverName]: { type: "remote", url: mcpUrl } }, | 61 | $schema: "https://opencode.ai/config.json", |
| 51 | ...(options?.chatBrainUrl | 62 | mcp: { [serverName]: { type: "remote", url: mcpUrl } }, |
| 52 | ? { | 63 | ...(options?.chatBrainUrl |
| 53 | autoupdate: false, | 64 | ? { |
| 54 | share: "disabled", | 65 | autoupdate: false, |
| 55 | model: "replay/replay-model", | 66 | share: "disabled", |
| 56 | permission: { "*": "allow" }, | 67 | model: "replay/replay-model", |
| 57 | provider: { | 68 | permission: { "*": "allow" }, |
| 58 | replay: { | 69 | provider: { |
| 59 | name: "Replay", | 70 | replay: { |
| 60 | npm: "@ai-sdk/openai-compatible", | 71 | name: "Replay", |
| 61 | options: { baseURL: options.chatBrainUrl, apiKey: "replay-key" }, | 72 | npm: "@ai-sdk/openai-compatible", |
| 62 | models: { "replay-model": { name: "Replay Model" } }, | 73 | options: { baseURL: options.chatBrainUrl, apiKey: "replay-key" }, |
| 74 | models: { "replay-model": { name: "Replay Model" } }, | ||
| 75 | }, | ||
| 63 | }, | 76 | }, |
| 64 | }, | 77 | } |
| 65 | } | 78 | : {}), |
| 66 | : {}), | 79 | }), |
| 67 | }), | 80 | ); |
| 68 | ); | 81 | } |
| 69 | // OpenCode launches the OAuth URL via `open`; the shim records it instead. | 82 | // OpenCode launches the OAuth URL via `open`; the shim records it instead. |
| 70 | writeFileSync(join(binDir, "open"), `#!/bin/sh\necho "$@" >> ${openedUrlsFile}\nexit 0\n`, { | 83 | writeFileSync(join(binDir, "open"), `#!/bin/sh\necho "$@" >> ${openedUrlsFile}\nexit 0\n`, { |
| 71 | mode: 0o755, | 84 | mode: 0o755, |
| @@ -134,3 +147,112 @@ export const completeOAuthConsent = async ( · 62 unmodified lines | |||
| 134 | } | 147 | } |
| 135 | throw new Error("opencode never opened an authorization URL"); | 148 | throw new Error("opencode never opened an authorization URL"); |
| 136 | }; | 149 | }; |
| 150 | |||
| 151 | // --------------------------------------------------------------------------- | ||
| 152 | // One-shot / multi-turn headless runs (`bun run cli opencode`) | ||
| 153 | // --------------------------------------------------------------------------- | ||
| 154 | |||
| 155 | const hostAuthFile = (): string => join(homedir(), ".local", "share", "opencode", "auth.json"); | ||
| 156 | |||
| 157 | export const hasOpenCodeCredential = (): boolean => existsSync(hostAuthFile()); | ||
| 158 | |||
| 159 | export interface OpenCodeRunOptions { | ||
| 160 | readonly prompt: string; | ||
| 161 | /** `provider/model`, e.g. `opencode/deepseek-v4-flash` (the default). */ | ||
| 162 | readonly model?: string; | ||
| 163 | /** Continue the home's most recent session (multi-turn). */ | ||
| 164 | readonly continueSession?: boolean; | ||
| 165 | /** Reuse a prior run's home (REQUIRED for continueSession to see it). */ | ||
| 166 | readonly home?: OpenCodeHome; | ||
| 167 | /** Expose one remote MCP server to the model (OpenCode's native client). | ||
| 168 | * OAuth, if the server needs it, is the caller's job (mcp auth + consent | ||
| 169 | * — see the mcp-opencode-real scenario). */ | ||
| 170 | readonly mcp?: { readonly serverName: string; readonly url: string }; | ||
| 171 | readonly timeoutMs?: number; | ||
| 172 | } | ||
| 173 | |||
| 174 | export interface OpenCodeRunResult { | ||
| 175 | readonly answerText: string; | ||
| 176 | /** Raw JSONL events (--format json), parsed, for grading/artifacts. */ | ||
| 177 | readonly events: readonly Record<string, unknown>[]; | ||
| 178 | readonly home: OpenCodeHome; | ||
| 179 | readonly exitCode: number | null; | ||
| 180 | readonly durationMs: number; | ||
| 181 | readonly stderr: string; | ||
| 182 | } | ||
| 183 | |||
| 184 | /** Drive the real OpenCode binary headless and hermetic: its own XDG home | ||
| 185 | * seeded with the machine's subscription credential. Pass `home` + | ||
| 186 | * `continueSession` for multi-turn; pass `mcp` to use OpenCode's native MCP | ||
| 187 | * client against one of our targets. */ | ||
| 188 | export const runOpenCode = async (options: OpenCodeRunOptions): Promise<OpenCodeRunResult> => { | ||
| 189 | if (!hasOpenCodeCredential()) { | ||
| 190 | throw new Error( | ||
| 191 | `opencode: no credential at ${hostAuthFile()} — run \`opencode auth login\` once on this machine`, | ||
| 192 | ); | ||
| 193 | } | ||
| 194 | const PLACEHOLDER_MCP = "http://127.0.0.1:9/"; | ||
| 195 | const home = | ||
| 196 | options.home ?? | ||
| 197 | makeOpenCodeHome(options.mcp?.serverName ?? "none", options.mcp?.url ?? PLACEHOLDER_MCP); | ||
| 198 | const configPath = join(home.projectDir, "opencode.json"); | ||
| 199 | if (!options.mcp && readFileSync(configPath, "utf8").includes(PLACEHOLDER_MCP)) { | ||
| 200 | // makeOpenCodeHome always writes an mcp block; a plain run wants none. | ||
| 201 | writeFileSync( | ||
| 202 | configPath, | ||
| 203 | JSON.stringify({ $schema: "https://opencode.ai/config.json", share: "disabled" }), | ||
| 204 | ); | ||
| 205 | } | ||
| 206 | // Seed the subscription credential into the hermetic home. | ||
| 207 | const authDir = join(home.env.XDG_DATA_HOME ?? "", "opencode"); | ||
| 208 | mkdirSync(authDir, { recursive: true }); | ||
| 209 | copyFileSync(hostAuthFile(), join(authDir, "auth.json")); | ||
| 210 | |||
| 211 | const args = [ | ||
| 212 | "run", | ||
| 213 | "-m", | ||
| 214 | options.model ?? "opencode/deepseek-v4-flash", | ||
| 215 | "--format", | ||
| 216 | "json", | ||
| 217 | ...(options.continueSession ? ["--continue"] : []), | ||
| 218 | options.prompt, | ||
| 219 | ]; | ||
| 220 | const startedAt = Date.now(); | ||
| 221 | const child = spawn("opencode", args, { | ||
| 222 | cwd: home.projectDir, | ||
| 223 | env: { ...home.env, PWD: home.projectDir }, | ||
| 224 | stdio: ["ignore", "pipe", "pipe"], | ||
| 225 | }); | ||
| 226 | let stdout = ""; | ||
| 227 | let stderr = ""; | ||
| 228 | child.stdout.on("data", (chunk: Buffer) => (stdout += chunk.toString("utf8"))); | ||
| 229 | child.stderr.on("data", (chunk: Buffer) => (stderr += chunk.toString("utf8"))); | ||
| 230 | const exitCode = await new Promise<number | null>((resolve) => { | ||
| 231 | const killer = setTimeout(() => child.kill("SIGKILL"), options.timeoutMs ?? 240_000); | ||
| 232 | child.once("exit", (code) => { | ||
| 233 | clearTimeout(killer); | ||
| 234 | resolve(code); | ||
| 235 | }); | ||
| 236 | }); | ||
| 237 | |||
| 238 | const events: Record<string, unknown>[] = []; | ||
| 239 | for (const line of stdout.split("\n")) { | ||
| 240 | if (!line.trim()) continue; | ||
| 241 | // oxlint-disable-next-line executor/no-try-catch-or-throw, executor/no-json-parse -- boundary: tolerant parse of opencode's JSONL stream | ||
| 242 | try { | ||
| 243 | events.push(JSON.parse(line) as Record<string, unknown>); | ||
| 244 | } catch { | ||
| 245 | // Non-JSON line — keep going. | ||
| 246 | } | ||
| 247 | } | ||
| 248 | const answerText = events | ||
| 249 | .filter( | ||
| 250 | (event) => | ||
| 251 | event.type === "text" && | ||
| 252 | typeof (event.part as { text?: string } | undefined)?.text === "string", | ||
| 253 | ) | ||
| 254 | .map((event) => (event.part as { text: string }).text) | ||
| 255 | .join("\n"); | ||
| 256 | |||
| 257 | return { answerText, events, home, exitCode, durationMs: Date.now() - startedAt, stderr }; | ||
| 258 | }; | ||
| @@ -7,17 +7,19 @@ · 6 unmodified lines | |||
| 7 | // credentials and without touching anyone's pi/OpenCode state. | 7 | // credentials and without touching anyone's pi/OpenCode state. |
| 8 | // | 8 | // |
| 9 | // Consumers: | 9 | // Consumers: |
| 10 | // - `bun run cli infer "..."` — the interactive command (scripts/cli.ts). | 10 | // - `bun run cli pi "..."` — the interactive command (scripts/cli.ts). |
| 11 | // - future eval tiers — fan out runAgent() trials and grade the results. | 11 | // - future eval tiers — fan out runPi() trials and grade the results. |
| 12 | // | 12 | // |
| 13 | // This is deliberately NOT the OpenCode binary: pi's headless mode is built | 13 | // This is deliberately NOT the OpenCode binary: pi's headless mode is built |
| 14 | // for this (clean JSONL events, --no-tools, hermetic config dir via env), | 14 | // for this (clean JSONL events, --no-tools, hermetic config dir via env), |
| 15 | // while OpenCode stays what it already is in this suite — the MCP-native | 15 | // while OpenCode stays what it already is in this suite — the MCP-native |
| 16 | // real-client actor (src/clients/opencode.ts). | 16 | // real-client actor (src/clients/opencode.ts). |
| 17 | import { spawn } from "node:child_process"; | 17 | import { spawn } from "node:child_process"; |
| 18 | import { createHash } from "node:crypto"; | ||
| 18 | import { mkdirSync, mkdtempSync, readFileSync, writeFileSync } from "node:fs"; | 19 | import { mkdirSync, mkdtempSync, readFileSync, writeFileSync } from "node:fs"; |
| 19 | import { homedir, tmpdir } from "node:os"; | 20 | import { homedir, tmpdir } from "node:os"; |
| 20 | import { join } from "node:path"; | 21 | import { join } from "node:path"; |
| 22 | import { fileURLToPath } from "node:url"; | ||
| 21 | 23 | ||
| 22 | // --------------------------------------------------------------------------- | 24 | // --------------------------------------------------------------------------- |
| 23 | // The subscription credential (OpenCode Zen — an OpenAI-compatible gateway) | 25 | // The subscription credential (OpenCode Zen — an OpenAI-compatible gateway) |
| @@ -57,19 +59,19 @@ export const DEFAULT_MODEL: string = ZEN_MODELS[0]; · 33 unmodified lines | |||
| 57 | // Result shape — distilled from pi's JSONL event stream | 59 | // Result shape — distilled from pi's JSONL event stream |
| 58 | // --------------------------------------------------------------------------- | 60 | // --------------------------------------------------------------------------- |
| 59 | 61 | ||
| 60 | export interface AgentToolCall { | 62 | export interface PiToolCall { |
| 61 | readonly name: string; | 63 | readonly name: string; |
| 62 | readonly args: unknown; | 64 | readonly args: unknown; |
| 63 | readonly result?: unknown; | 65 | readonly result?: unknown; |
| 64 | readonly isError?: boolean; | 66 | readonly isError?: boolean; |
| 65 | } | 67 | } |
| 66 | 68 | ||
| 67 | export interface AgentRunResult { | 69 | export interface PiRunResult { |
| 68 | /** All assistant text parts of the final answer, joined. */ | 70 | /** All assistant text parts of the final answer, joined. */ |
| 69 | readonly answerText: string; | 71 | readonly answerText: string; |
| 70 | /** The model's reasoning text, when the model emits it. */ | 72 | /** The model's reasoning text, when the model emits it. */ |
| 71 | readonly thinkingText: string; | 73 | readonly thinkingText: string; |
| 72 | readonly toolCalls: readonly AgentToolCall[]; | 74 | readonly toolCalls: readonly PiToolCall[]; |
| 73 | /** Every JSONL event pi emitted, parsed, in order — for grading/artifacts. */ | 75 | /** Every JSONL event pi emitted, parsed, in order — for grading/artifacts. */ |
| 74 | readonly events: readonly Record<string, unknown>[]; | 76 | readonly events: readonly Record<string, unknown>[]; |
| 75 | readonly usage?: { readonly input: number; readonly output: number }; | 77 | readonly usage?: { readonly input: number; readonly output: number }; |
| @@ -79,7 +81,7 @@ export interface AgentRunResult { · 3 unmodified lines | |||
| 79 | readonly stderr: string; | 81 | readonly stderr: string; |
| 80 | } | 82 | } |
| 81 | 83 | ||
| 82 | export interface AgentRunOptions { | 84 | export interface PiRunOptions { |
| 83 | readonly prompt: string; | 85 | readonly prompt: string; |
| 84 | /** Zen model id (see ZEN_MODELS). Default: cheap + fast. */ | 86 | /** Zen model id (see ZEN_MODELS). Default: cheap + fast. */ |
| 85 | readonly model?: string; | 87 | readonly model?: string; |
| @@ -91,11 +93,25 @@ export interface AgentRunOptions { · 5 unmodified lines | |||
| 91 | readonly cwd?: string; | 93 | readonly cwd?: string; |
| 92 | /** Replace pi's coding system prompt. */ | 94 | /** Replace pi's coding system prompt. */ |
| 93 | readonly systemPrompt?: string; | 95 | readonly systemPrompt?: string; |
| 96 | /** Resume/create a named persistent session (multi-turn). Sessions live in | ||
| 97 | * `sessionDir` (default e2e/.dev/pi-sessions), so a follow-up call with | ||
| 98 | * the same name continues the conversation. Omit for a one-shot. */ | ||
| 99 | readonly session?: string; | ||
| 100 | readonly sessionDir?: string; | ||
| 94 | readonly timeoutMs?: number; | 101 | readonly timeoutMs?: number; |
| 95 | } | 102 | } |
| 96 | 103 | ||
| 97 | const DEFAULT_TIMEOUT_MS = 120_000; | 104 | const DEFAULT_TIMEOUT_MS = 120_000; |
| 98 | 105 | ||
| 106 | const defaultSessionDir = (): string => | ||
| 107 | fileURLToPath(new URL("../../.dev/pi-sessions/", import.meta.url)); | ||
| 108 | |||
| 109 | /** pi wants a UUID session id; derive one stably from the friendly name. */ | ||
| 110 | const sessionUuid = (name: string): string => { | ||
| 111 | const hex = createHash("sha256").update(name).digest("hex"); | ||
| 112 | return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-4${hex.slice(13, 16)}-8${hex.slice(17, 20)}-${hex.slice(20, 32)}`; | ||
| 113 | }; | ||
| 114 | |||
| 99 | // --------------------------------------------------------------------------- | 115 | // --------------------------------------------------------------------------- |
| 100 | // Run | 116 | // Run |
| 101 | // --------------------------------------------------------------------------- | 117 | // --------------------------------------------------------------------------- |
| @@ -121,7 +137,7 @@ const providerExtension = (modelIds: readonly string[]): string => `export defau · 19 unmodified lines | |||
| 121 | } | 137 | } |
| 122 | `; | 138 | `; |
| 123 | 139 | ||
| 124 | export const runAgent = async (options: AgentRunOptions): Promise<AgentRunResult> => { | 140 | export const runPi = async (options: PiRunOptions): Promise<PiRunResult> => { |
| 125 | const key = zenApiKey(); | 141 | const key = zenApiKey(); |
| 126 | if (!key) { | 142 | if (!key) { |
| 127 | throw new Error( | 143 | throw new Error( |
| @@ -142,7 +158,23 @@ export const runAgent = async (options: AgentRunOptions): Promise<AgentRunResult · 14 unmodified lines | |||
| 142 | ? ZEN_MODELS | 158 | ? ZEN_MODELS |
| 143 | : [...ZEN_MODELS, model]; | 159 | : [...ZEN_MODELS, model]; |
| 144 | writeFileSync(extensionPath, providerExtension(modelIds)); | 160 | writeFileSync(extensionPath, providerExtension(modelIds)); |
| 145 | const cwd = options.cwd ?? join(scratch, "project"); | 161 | |
| 162 | // Named sessions persist (and resume) under sessionDir; one-shots don't. | ||
| 163 | // pi session ids are PROJECT-scoped, so a named session also pins its cwd — | ||
| 164 | // otherwise every call's fresh scratch dir would start a new conversation. | ||
| 165 | const sessionRoot = options.session | ||
| 166 | ? join(options.sessionDir ?? defaultSessionDir(), options.session) | ||
| 167 | : undefined; | ||
| 168 | const sessionArgs = sessionRoot | ||
| 169 | ? [ | ||
| 170 | "--session-id", | ||
| 171 | sessionUuid(options.session!), | ||
| 172 | "--session-dir", | ||
| 173 | join(sessionRoot, "sessions"), | ||
| 174 | ] | ||
| 175 | : ["--no-session"]; | ||
| 176 | const cwd = | ||
| 177 | options.cwd ?? (sessionRoot ? join(sessionRoot, "project") : join(scratch, "project")); | ||
| 146 | mkdirSync(cwd, { recursive: true }); | 178 | mkdirSync(cwd, { recursive: true }); |
| 147 | 179 | ||
| 148 | const args = [ | 180 | const args = [ |
| @@ -152,7 +184,7 @@ export const runAgent = async (options: AgentRunOptions): Promise<AgentRunResult · 3 unmodified lines | |||
| 152 | "zen", | 184 | "zen", |
| 153 | "--model", | 185 | "--model", |
| 154 | model, | 186 | model, |
| 155 | "--no-session", | 187 | ...sessionArgs, |
| 156 | "--mode", | 188 | "--mode", |
| 157 | "json", | 189 | "json", |
| 158 | ...(options.tools ? [] : ["--no-tools"]), | 190 | ...(options.tools ? [] : ["--no-tools"]), |
| @@ -230,7 +262,7 @@ interface AgentEndEvent { · 71 unmodified lines | |||
| 230 | 262 | ||
| 231 | const distill = ( | 263 | const distill = ( |
| 232 | jsonl: string, | 264 | jsonl: string, |
| 233 | ): Pick<AgentRunResult, "answerText" | "thinkingText" | "toolCalls" | "events" | "usage"> => { | 265 | ): Pick<PiRunResult, "answerText" | "thinkingText" | "toolCalls" | "events" | "usage"> => { |
| 234 | const events: Record<string, unknown>[] = []; | 266 | const events: Record<string, unknown>[] = []; |
| 235 | for (const line of jsonl.split("\n")) { | 267 | for (const line of jsonl.split("\n")) { |
| 236 | if (!line.trim()) continue; | 268 | if (!line.trim()) continue; |
| @@ -242,10 +274,10 @@ const distill = ( · 5 unmodified lines | |||
| 242 | } | 274 | } |
| 243 | } | 275 | } |
| 244 | 276 | ||
| 245 | const toolCalls = new Map<string, AgentToolCall>(); | 277 | const toolCalls = new Map<string, PiToolCall>(); |
| 246 | let answerText = ""; | 278 | let answerText = ""; |
| 247 | let thinkingText = ""; | 279 | let thinkingText = ""; |
| 248 | let usage: AgentRunResult["usage"]; | 280 | let usage: PiRunResult["usage"]; |
| 249 | 281 | ||
| 250 | for (const event of events as readonly AgentEndEvent[]) { | 282 | for (const event of events as readonly AgentEndEvent[]) { |
| 251 | if (event.type === "tool_execution_start" && event.toolCallId) { | 283 | if (event.type === "tool_execution_start" && event.toolCallId) { |