History

Developer-session e2e: chat-theater recordings, focus timeline, one-mp4 films (#962)

* Developer-session e2e: real OpenCode chat on camera via a replay brain A scenario now runs the way a developer actually tests the product: chat with the real OpenCode TUI in a recorded PTY, the agent registers an API and drops a connect link in the chat (real MCP tool execution), a browser opens the link and pastes a real emulator-minted key while the terminal session stays open, and the chat resumes to verify the connection with a live send confirmed by the provider emulator's request ledger. - e2e/src/clients/replay-brain.ts: local OpenAI-wire chat-completions server (SSE + tool calls) scripted per scenario by transcript inspection, so approval pauses and OpenCode side requests don't derail it. Tool names resolve by suffix against whatever the agent offers. - clients/opencode.ts: makeOpenCodeHome optionally wires a 'replay' provider (openai-compatible) at the brain and pre-allows permissions. - scenarios/connect-handoff-session.test.ts: one continuous terminal.cast timeline with the browser hop forked concurrently; chat-brain.json transcript written as a run artifact even on failure. * Chat theater: agent-chat recordings over real mcporter calls, no inference The developer-session recording no longer needs a third-party agent binary or a faked LLM: the 'agent' is a chat presentation (agent-chat-tui.ts, a renderer in the recorded PTY) driven by chat-theater.ts while the scenario makes REAL mcporter MCP calls — OAuth, execute, approval pause/resume all genuine, each on-screen tool spinner bracketing the actual call it narrates. Runs anywhere bun runs (no opencode capability), ~28s on cloud. replay-brain.ts stays for the distinct tier where the third-party client's own protocol behavior is under test (OpenCode/Claude Code); real-inference evals remain a separate axis (performance distributions, not pass/fail scenarios). * Viewer: a run with both recordings plays as one session in time order Runs from chat-theater scenarios produce a terminal.cast (the chat) AND a session.mp4 (the browser hop). The run page previously picked one (video ?? cast), hiding the chat that led to the browser. Now both render under a single 'session' tab in story order — 1 · the chat, in the terminal; 2 · the browser hop, mid-chat — with autoplay reserved for video-only runs. * The Desk: film a scenario on one virtual desktop, one recording e2e/desk/run.sh builds a Linux image (Xvfb + openbox + xterm + headed Chromium + ffmpeg) and runs a scenario inside it with E2E_DESK=1: the chat renderer opens in a visible UTF-8 xterm (events over a FIFO instead of a PTY), the browser hop is a real headed window raised over it, and a single x11grab films the whole screen for the whole run — no splicing. The film replaces session.mp4 in the scenario's run dir; the scenario file itself is unchanged (chat-theater switches transports on E2E_DESK). First run builds the image and fills a node_modules volume; subsequent films of the connect-handoff session take ~2 minutes end to end. * film.ts: the session as one mp4 — full-screen cuts, no virtual desktop bun scripts/film.ts runs/<target>/<slug> turns a chat-theater run's two real recordings into one video that plays like a screen capture of a developer tabbing between full-screen windows: the cast (rendered without idle compression so cast time == video time) is cut at the browser hop — located by the scenario's narrator line, falling back to the largest output gap — and the browser recording fills the gap. Both segments are the genuine recordings; only the cut is editorial, exactly like tabbing. The film registers itself in result.json and the viewer plays film.mp4 as the session when present. Runs anywhere (agg + ffmpeg) — no Docker, no display server; the Desk remains for when actual window layering matters. * Focus is derived from operations: surfaces emit the run's focus timeline Acting on a window IS focusing it: a Playwright step marks the browser focused, a chat-theater event marks the terminal focused. Surfaces record these transitions (plus each recording's start anchor) into the run's timeline.json as a side effect of normal driving — no scenario changes, no declared focus anywhere. film.ts now builds its cut list from that timeline (any number of hops, each act mapped into its recording's own clock via the anchors), keeping the narrator-line/largest-gap heuristic only as a fallback for runs without one. The film is the operations, edited by where they happened. * Developer-session scenario adds Resend by served spec URL only The emulator serves its own OpenAPI document, so the scenario stops hand-writing a Resend subset and an explicit auth template — addSpec gets just the spec URL, exactly what an agent would send, and the platform derives base URL and the paste-a-token method from the spec itself. * Chat theater tool calls render like a real agent TUI chat.tool now takes the call itself — tool name, the REAL input (the sandbox code the scenario executes, shown as the indented preview block agent TUIs draw), and a result-line renderer — instead of a prose label. The renderer paints name → input block → live spinner → ✓ with the real duration and the real result's first line. The recording reads like an actual agent session: what was called, with what, how long it took, what came back. * pr-media prefers film.mp4 — the whole session, not just the browser hop * Auto-film runs that produce both recordings film.mp4 was a manual post-step, so any re-run left the viewer showing the parts (cast + browser video) instead of the spliced session. The runner now invokes scripts/film.ts after every successful run that has both terminal.cast and session.mp4 — best-effort, so missing agg/ffmpeg degrades back to the parts rather than failing the run. Every developer- session run's page now plays the single film by default. * Pin vendor/emulate at the OpenAPI-serving emulators Every REST emulator now serves its own OpenAPI document; the connect scenarios register integrations from those live spec URLs instead of hand-written specs.

Rhys Sullivan committed 5 months ago

Commit fe5181b1 parent f15f33d16 files · +1232 −15Browse files

Unified · Split

16 files
  1. e2e/AGENTS.md +29 −0
  2. e2e/desk/Dockerfile +27 −0
  3. e2e/desk/entry.sh +43 −0
  4. e2e/desk/run.sh +17 −0
  5. e2e/scenarios/connect-handoff-session.test.ts +189 −0
  6. e2e/scripts/film.ts +193 −0
  7. e2e/scripts/pr-media.ts +8 −4
  8. e2e/src/clients/agent-chat-tui.ts +166 −0
  9. e2e/src/clients/chat-theater.ts +217 −0
  10. e2e/src/clients/opencode.ts +28 −2
  11. e2e/src/clients/replay-brain.ts +208 −0
  12. e2e/src/scenario.ts +25 −1
  13. e2e/src/surfaces/browser.ts +17 −1
  14. e2e/src/timeline.ts +47 −0
  15. e2e/viewer/src/App.tsx +17 −6
  16. vendor/emulate +1 −1
e2e/AGENTS.md+29−0
@@ -133,3 +133,32 @@ When handing results to the user, follow the evidence contract in the root · 132 unmodified lines
133133 values, not booleans — `expect(list).toContain(x)`, never
134134 `expect(list.includes(x)).toBe(true)` — so failures show the data.
135135 - Keep it deterministic: no sleeps; wait on conditions.
136+
137+## Developer-session recordings (chat theater + desk)
138+
139+Some scenarios are meant to be WATCHED — they show the product the way a
140+developer actually uses it. Three tiers, pick deliberately:
141+
142+1. **Chat theater** (`src/clients/chat-theater.ts`): the default for
143+ product-flow recordings. The "agent" is a chat renderer in a recorded
144+ PTY; every tool spinner brackets a REAL mcporter MCP call (OAuth,
145+ execute, approval resume). No inference, no third-party binary.
146+ Exemplar: `scenarios/connect-handoff-session.test.ts`. Artifacts:
147+ `terminal.cast` (the chat) + `session.mp4` (browser hops); the viewer
148+ plays them in story order.
149+2. **Replay brain + real client** (`src/clients/replay-brain.ts`): when the
150+ third-party CLIENT's behavior is under test (OpenCode/Claude Code
151+ protocol handling). A scripted OpenAI-wire server plays the LLM; the
152+ real client does everything else. Script by transcript inspection, never
153+ turn counting.
154+3. **Real-inference evals**: a different axis (performance distributions,
155+ not pass/fail). Not in this suite.
156+
157+**The Desk** (`desk/`): films a scenario on one virtual Linux desktop — the
158+chat renderer in a visible xterm, the browser as a real headed window, one
159+ffmpeg x11grab. The film replaces session.mp4 in the run dir; the scenario
160+file is unchanged (chat-theater switches transports on `E2E_DESK=1`).
161+
162+```
163+e2e/desk/run.sh [scenario] [project] # docker; first run builds + installs
164+```
e2e/desk/Dockerfile+27−0
@@ -0,0 +1,27 @@
1+# The Desk: a virtual Linux desktop that e2e scenarios are FILMED on — one
2+# Xvfb display showing a real terminal window (the chat) and a real headed
3+# Chromium (the browser hop), captured by a single ffmpeg x11grab into one
4+# session.mp4. No splicing: the film is the screen.
5+#
6+# The repo is bind-mounted at /repo at runtime; node_modules and caches live
7+# in named volumes (host artifacts are macOS-native and can't be reused).
8+FROM node:22-bookworm
9+
10+RUN apt-get update && apt-get install -y --no-install-recommends \
11+ xvfb openbox xterm ffmpeg xdotool x11-utils x11-xserver-utils \
12+ fonts-dejavu-core fonts-noto-color-emoji \
13+ curl unzip ca-certificates git \
14+ && rm -rf /var/lib/apt/lists/*
15+
16+# Bun — the repo's package manager and script runner.
17+RUN curl -fsSL https://bun.sh/install | bash -s "bun-v1.3.13" \
18+ && ln -s /root/.bun/bin/bun /usr/local/bin/bun \
19+ && ln -s /root/.bun/bin/bun /usr/local/bin/bunx
20+
21+# Chromium + system deps baked into the image, pinned to the repo's
22+# playwright version so the runtime install check is a no-op.
23+ENV PLAYWRIGHT_BROWSERS_PATH=/ms-playwright
24+RUN npx -y playwright@1.60.0 install --with-deps chromium && rm -rf /root/.npm
25+
26+WORKDIR /repo
27+ENTRYPOINT ["bash", "e2e/desk/entry.sh"]
e2e/desk/entry.sh+43−0
@@ -0,0 +1,43 @@
1+#!/usr/bin/env bash
2+# Desk entrypoint (runs inside the desk image): boot a display, roll one
3+# camera on it, run the scenario with E2E_DESK=1 so its windows land on the
4+# display, then drop the film into the scenario's run dir as session.mp4.
5+set -euo pipefail
6+
7+SCENARIO="${DESK_SCENARIO:-scenarios/connect-handoff-session.test.ts}"
8+PROJECT="${DESK_PROJECT:-cloud}"
9+SIZE="${DESK_SIZE:-1440x900}"
10+
11+echo "[desk] installing dependencies (cached in a volume after first run)…"
12+bun install --frozen-lockfile
13+
14+echo "[desk] display :99 at ${SIZE}"
15+Xvfb :99 -screen 0 "${SIZE}x24" -nolisten tcp &
16+export DISPLAY=:99
17+sleep 0.5
18+openbox &
19+xsetroot -solid "#0b0b10" || true
20+
21+mkdir -p /desk-out
22+echo "[desk] camera rolling"
23+ffmpeg -loglevel error -f x11grab -framerate 24 -video_size "$SIZE" -i :99 \
24+ -c:v libx264 -preset veryfast -crf 24 -pix_fmt yuv420p -y /desk-out/desk.mp4 &
25+FFMPEG_PID=$!
26+
27+set +e
28+(cd e2e && E2E_DESK=1 npx vitest run --project "$PROJECT" "$SCENARIO")
29+STATUS=$?
30+set -e
31+
32+sleep 1
33+kill -INT "$FFMPEG_PID" 2>/dev/null || true
34+wait "$FFMPEG_PID" 2>/dev/null || true
35+
36+# The desk film IS the session recording — it replaces the browser-only
37+# video the surface recorded inside the run.
38+RUN_DIR=$(ls -dt e2e/runs/"$PROJECT"/*/ 2>/dev/null | head -1)
39+if [ -n "$RUN_DIR" ] && [ -f /desk-out/desk.mp4 ]; then
40+ cp /desk-out/desk.mp4 "${RUN_DIR}session.mp4"
41+ echo "[desk] film → ${RUN_DIR}session.mp4"
42+fi
43+exit "$STATUS"
e2e/desk/run.sh+17−0
@@ -0,0 +1,17 @@
1+#!/usr/bin/env bash
2+# Film a scenario on the Desk (one virtual desktop, one recording).
3+# e2e/desk/run.sh [scenario-path] [project]
4+# First run builds the image and fills the node_modules volume; later runs
5+# reuse both.
6+set -euo pipefail
7+cd "$(dirname "$0")/../.."
8+
9+docker build -t executor-e2e-desk e2e/desk
10+
11+docker run --rm \
12+ -v "$PWD":/repo \
13+ -v executor-desk-node-modules:/repo/node_modules \
14+ -v executor-desk-bun-cache:/root/.bun/install/cache \
15+ -e DESK_SCENARIO="${1:-scenarios/connect-handoff-session.test.ts}" \
16+ -e DESK_PROJECT="${2:-cloud}" \
17+ executor-e2e-desk
e2e/scenarios/connect-handoff-session.test.ts+189−0
@@ -0,0 +1,189 @@
1+// The connect handoff as a DEVELOPER SESSION — the way a human actually
2+// tests this: an agent chat in a real terminal where the agent wires up the
3+// API over MCP and drops a connect link, a browser hop to paste the key,
4+// then back to the chat to prove the connection works with a live send.
5+//
6+// No inference, no third-party agent binary: the "agent" is the chat
7+// theater (src/clients/chat-theater.ts) presenting REAL mcporter MCP calls
8+// — OAuth, execute, approval pause/resume all genuine, every tool spinner
9+// on screen bracketing the actual call it narrates. The provider on the
10+// other side is real too (resend.emulators.dev); its request ledger is the
11+// final evidence.
12+import { randomBytes } from "node:crypto";
13+import { join } from "node:path";
14+
15+import { expect } from "@effect/vitest";
16+import { Effect } from "effect";
17+
18+import { scenario } from "../src/scenario";
19+import { Browser, Cli, Mcp, RunDir, Target } from "../src/services";
20+import { withChatTheater } from "../src/clients/chat-theater";
21+import type { McpSession } from "../src/surfaces/mcp";
22+
23+const EMULATOR_BASE = "https://resend.emulators.dev";
24+
25+const unique = (prefix: string) => `${prefix}_${randomBytes(4).toString("hex")}`;
26+
27+// The emulator serves its own OpenAPI document (bearer auth, base URL in
28+// `servers`) — adding by URL with nothing else is exactly what an agent
29+// does, and the platform derives the paste-a-token auth method from the
30+// spec's security scheme.
31+const EMULATOR_SPEC_URL = `${EMULATOR_BASE}/openapi.json`;
32+
33+const addSpecCode = (slug: string) => `
34+const added = await tools.executor.openapi.addSpec({
35+ spec: { kind: "url", url: ${JSON.stringify(EMULATOR_SPEC_URL)} },
36+ slug: ${JSON.stringify(slug)},
37+});
38+return added.ok ? { ok: true, slug: added.data.slug, toolCount: added.data.toolCount } : { ok: false, error: added.error };
39+`;
40+
41+const createHandoffCode = (slug: string) => `
42+const handoff = await tools.executor.coreTools.connections.createHandoff({
43+ integration: ${JSON.stringify(slug)},
44+ owner: "org",
45+ label: "Resend",
46+});
47+return handoff.ok ? { ok: true, url: handoff.data.url } : { ok: false, error: handoff.error };
48+`;
49+
50+const sendEmailCode = (slug: string, subject: string) => `
51+const found = await tools.search({ namespace: ${JSON.stringify(slug)}, query: "send email", limit: 5 });
52+const path = found.items[0]?.path;
53+if (!path) return { ok: false, error: "no send tool found" };
54+let t = tools;
55+for (const seg of path.split(".")) t = t[seg];
56+const sent = await t({
57+ body: {
58+ from: "onboarding@example.com",
59+ to: "dev-session@example.com",
60+ subject: ${JSON.stringify(subject)},
61+ html: "<p>connect-handoff developer session</p>",
62+ },
63+});
64+return { ok: sent.ok, path, result: sent.ok ? sent.data : sent.error };
65+`;
66+
67+/** Run `execute`, auto-approving a paused execution (policy elicitation)
68+ * once, and parse the sandbox's JSON return value. */
69+const executeJson = (session: McpSession, code: string) =>
70+ Effect.gen(function* () {
71+ let result = yield* session.call("execute", { code });
72+ if (result.text.includes("executionId:")) {
73+ result = yield* session.approvePaused(result.text);
74+ }
75+ expect(result.ok, `execute completed (got: ${result.text.slice(0, 400)})`).toBe(true);
76+ return JSON.parse(result.text) as Record<string, unknown>;
77+ });
78+
79+const mintEmulatorApiKey = Effect.promise(async () => {
80+ const response = await fetch(`${EMULATOR_BASE}/_emulate/credentials`, {
81+ method: "POST",
82+ headers: { "content-type": "application/json" },
83+ body: JSON.stringify({ type: "api-key" }),
84+ });
85+ const body = (await response.json()) as { credential?: { token?: string } };
86+ const token = body.credential?.token;
87+ if (!token) throw new Error(`emulator credential mint failed: ${JSON.stringify(body)}`);
88+ return token;
89+});
90+
91+scenario(
92+ "Connect · developer session: agent chat → handoff link → paste key → verified send",
93+ { timeout: 240_000 },
94+ Effect.scoped(
95+ Effect.gen(function* () {
96+ const target = yield* Target;
97+ const mcp = yield* Mcp;
98+ const browser = yield* Browser;
99+ const cli = yield* Cli;
100+ const runDir = yield* RunDir;
101+
102+ const integration = unique("resendsesh");
103+ const emailSubject = unique("dev-session");
104+ const apiKey = yield* mintEmulatorApiKey;
105+ const identity = yield* target.newIdentity();
106+ const session = mcp.session(identity);
107+
108+ yield* withChatTheater(
109+ cli,
110+ { title: "executor agent — connect Resend", record: join(runDir, "terminal.cast") },
111+ (chat) =>
112+ Effect.gen(function* () {
113+ // Real MCP OAuth + tool discovery happens behind this call.
114+ yield* chat.tool(
115+ { name: "executor (mcp)", result: (tools) => `${tools.length} tools available` },
116+ session.listTools(),
117+ );
118+
119+ yield* chat.user(
120+ "Add the Resend API to my executor and give me a link to connect my account",
121+ );
122+ yield* chat.assistant("I'll register the Resend API in your Executor now.");
123+ const added = yield* chat.tool(
124+ { name: "execute", input: addSpecCode(integration) },
125+ executeJson(session, addSpecCode(integration)),
126+ );
127+ expect(added.ok, `addSpec succeeded: ${JSON.stringify(added)}`).toBe(true);
128+
129+ yield* chat.assistant("Registered. Creating your connect link…");
130+ const handoff = yield* chat.tool(
131+ { name: "execute", input: createHandoffCode(integration) },
132+ executeJson(session, createHandoffCode(integration)),
133+ );
134+ expect(handoff.ok, `createHandoff succeeded: ${JSON.stringify(handoff)}`).toBe(true);
135+ const handoffUrl = String(handoff.url);
136+ expect(new URL(handoffUrl).origin, "handoff targets this deployment").toBe(
137+ new URL(target.baseUrl).origin,
138+ );
139+
140+ yield* chat.assistant(
141+ `Open this link to connect your Resend account:\n\n${handoffUrl}\n\nTell me once you've pasted your API key.`,
142+ );
143+
144+ // The browser hop — the terminal session stays open while the
145+ // "user" pastes the key; the paste is the real add-account UI.
146+ yield* chat.status("you, in the browser: opening the link and pasting the API key…");
147+ yield* browser.session(identity, async ({ page, step }) => {
148+ await step("Open the connect link from the chat", async () => {
149+ await page.goto(handoffUrl, { waitUntil: "networkidle" });
150+ await page
151+ .getByRole("heading", { name: /Add connection/ })
152+ .waitFor({ timeout: 15_000 });
153+ });
154+ await step("Paste the Resend API key and connect", async () => {
155+ const credential = page.getByPlaceholder(/paste the value \/ token/i);
156+ await credential.waitFor({ timeout: 15_000 });
157+ await credential.fill(apiKey);
158+ await page.getByRole("button", { name: "Add connection", exact: true }).click();
159+ await page
160+ .getByRole("heading", { name: /Add connection/ })
161+ .waitFor({ state: "hidden", timeout: 20_000 });
162+ });
163+ });
164+
165+ yield* chat.user("Connected, now send a test email to prove it works");
166+ yield* chat.assistant("Sending a test email through your new connection…");
167+ const sent = yield* chat.tool(
168+ { name: "execute", input: sendEmailCode(integration, emailSubject) },
169+ executeJson(session, sendEmailCode(integration, emailSubject)),
170+ );
171+ expect(sent.ok, `email sent through the connection: ${JSON.stringify(sent)}`).toBe(
172+ true,
173+ );
174+
175+ yield* chat.assistant("Test email sent - your Resend connection works.");
176+ }),
177+ );
178+
179+ // Final evidence: the emulator's ledger saw the send from Executor.
180+ const ledger = yield* Effect.promise(async () =>
181+ (await fetch(`${EMULATOR_BASE}/_emulate/ledger`)).text(),
182+ );
183+ expect(
184+ ledger.includes(emailSubject),
185+ "the emulator request ledger recorded the test email",
186+ ).toBe(true);
187+ }),
188+ ),
189+);
e2e/scripts/film.ts+193−0
@@ -0,0 +1,193 @@
1+// Film a run: turn a chat-theater run's real recordings into ONE mp4 that
2+// plays like a screen recording of a developer tabbing between full-screen
3+// windows — terminal, browser, terminal. Both segments are the genuine
4+// recordings; the only editorial act is the cut, exactly like real tabbing.
5+//
6+// bun scripts/film.ts runs/<target>/<slug>
7+//
8+// The cut list comes from the run's focus timeline (src/timeline.ts):
9+// surfaces mark focus as a side effect of acting — a Playwright step
10+// focuses the browser, a chat event focuses the terminal — so the
11+// operations themselves decide where the film cuts, for any number of
12+// hops. Runs without a timeline fall back to the narrator-line / largest-
13+// gap heuristic. The cast renders with NO idle compression so cast time
14+// equals video time. Output: film.mp4, registered in result.json — the
15+// viewer plays it as the session when present.
16+import { execFileSync } from "node:child_process";
17+import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
18+import { tmpdir } from "node:os";
19+import { join, resolve } from "node:path";
20+
21+import { readTimeline } from "../src/timeline";
22+
23+const runDir = resolve(process.argv[2] ?? "");
24+const castPath = join(runDir, "terminal.cast");
25+const browserPath = join(runDir, "session.mp4");
26+if (!existsSync(castPath) || !existsSync(browserPath)) {
27+ console.error(`film: need terminal.cast + session.mp4 in ${runDir}`);
28+ process.exit(1);
29+}
30+
31+const run = (cmd: string, args: string[]) => execFileSync(cmd, args, { stdio: "pipe" });
32+const probeSeconds = (file: string): number =>
33+ Number(
34+ execFileSync("ffprobe", [
35+ ...["-v", "quiet", "-show_entries", "format=duration"],
36+ ...["-of", "csv=p=0", file],
37+ ])
38+ .toString()
39+ .trim(),
40+ );
41+
42+// ---------------------------------------------------------------------------
43+// Build the cut list: acts of { source, from, to } in each source's clock.
44+// ---------------------------------------------------------------------------
45+
46+interface CastEvent {
47+ readonly at: number;
48+ readonly text: string;
49+}
50+
51+const events: CastEvent[] = readFileSync(castPath, "utf8")
52+ .split("\n")
53+ .slice(1)
54+ .filter(Boolean)
55+ .map((line) => JSON.parse(line) as [number, string, string])
56+ .filter((event) => event[1] === "o")
57+ .map((event) => ({ at: event[0], text: event[2] }));
58+
59+const castEnd = events.at(-1)?.at ?? 0;
60+
61+interface Act {
62+ readonly source: "terminal" | "browser";
63+ readonly from: number;
64+ readonly to: number;
65+}
66+
67+/** Acts from the focus timeline: each contiguous focus run becomes a cut of
68+ * that window's recording, mapped via the recording-start anchors. */
69+const actsFromTimeline = (): Act[] | null => {
70+ const timeline = readTimeline(runDir);
71+ if (!timeline || timeline.focus.length < 2) return null;
72+ const terminalAnchor = timeline.anchors.terminal;
73+ const browserAnchor = timeline.anchors.browser;
74+ if (terminalAnchor === undefined || browserAnchor === undefined) return null;
75+
76+ const browserEnd = probeSeconds(browserPath);
77+ const toMedia = (window: Act["source"], wallMs: number) =>
78+ Math.max(0, (wallMs - (window === "terminal" ? terminalAnchor : browserAnchor)) / 1000);
79+
80+ const acts: Act[] = [];
81+ for (let i = 0; i < timeline.focus.length; i += 1) {
82+ const current = timeline.focus[i];
83+ if (!current) continue;
84+ const nextAt = timeline.focus[i + 1]?.at;
85+ const from = i === 0 && current.window === "terminal" ? 0 : toMedia(current.window, current.at);
86+ const mediaEnd = current.window === "terminal" ? castEnd : browserEnd;
87+ const to =
88+ nextAt === undefined ? mediaEnd : Math.min(toMedia(current.window, nextAt), mediaEnd);
89+ if (to - from > 0.5) acts.push({ source: current.window, from, to });
90+ }
91+ return acts.length >= 2 ? acts : null;
92+};
93+
94+/** Fallback for runs without a timeline: one browser hop located by the
95+ * narrator line, or the largest output gap in the cast. */
96+const actsFromHeuristic = (): Act[] => {
97+ const findHop = (): { start: number; end: number } => {
98+ const markerIndex = events.findIndex((event) => event.text.includes("in the browser"));
99+ if (markerIndex !== -1) {
100+ const start = events[markerIndex];
101+ const after = events
102+ .slice(markerIndex + 1)
103+ .find((event) => event.at > (start?.at ?? 0) + 1 && event.text.trim().length > 0);
104+ if (start && after) return { start: start.at + 0.8, end: after.at };
105+ }
106+ let best = { start: 0, end: 0 };
107+ for (let i = 1; i < events.length; i += 1) {
108+ const previous = events[i - 1];
109+ const current = events[i];
110+ if (previous && current && current.at - previous.at > best.end - best.start) {
111+ best = { start: previous.at, end: current.at };
112+ }
113+ }
114+ return best;
115+ };
116+ const hop = findHop();
117+ if (hop.end - hop.start < 2) {
118+ console.error(`film: no browser hop found in the cast (gap ${hop.end - hop.start}s)`);
119+ process.exit(1);
120+ }
121+ return [
122+ { source: "terminal", from: 0, to: hop.start },
123+ { source: "browser", from: 0, to: probeSeconds(browserPath) },
124+ { source: "terminal", from: hop.end, to: castEnd },
125+ ];
126+};
127+
128+const acts = actsFromTimeline() ?? actsFromHeuristic();
129+
130+// ---------------------------------------------------------------------------
131+// Render + cut + concatenate onto one canvas — full-screen cuts, like
132+// tabbing.
133+// ---------------------------------------------------------------------------
134+
135+const work = mkdtempSync(join(tmpdir(), "e2e-film-"));
136+const castGif = join(work, "cast.gif");
137+const castVideo = join(work, "cast.mp4");
138+
139+run("agg", [
140+ ...["--idle-time-limit", String(Math.ceil(castEnd) + 60)],
141+ ...["--font-size", "16"],
142+ castPath,
143+ castGif,
144+]);
145+run("ffmpeg", [
146+ ...["-y", "-i", castGif],
147+ // agg's gif can have odd dimensions; libx264 requires even.
148+ ...["-vf", "scale=ceil(iw/2)*2:ceil(ih/2)*2"],
149+ ...["-pix_fmt", "yuv420p", "-r", "24", castVideo],
150+]);
151+
152+const FIT =
153+ "scale=1280:800:force_original_aspect_ratio=decrease,pad=1280:800:(ow-iw)/2:(oh-ih)/2:color=0x0b0b10,setsar=1,fps=24,format=yuv420p";
154+
155+const filmPath = join(runDir, "film.mp4");
156+const inputIndex = { terminal: 0, browser: 1 } as const;
157+const filters = acts.map(
158+ (act, index) =>
159+ `[${inputIndex[act.source]}:v]trim=${act.from.toFixed(2)}:${act.to.toFixed(2)},setpts=PTS-STARTPTS,${FIT}[act${index}]`,
160+);
161+run("ffmpeg", [
162+ "-y",
163+ ...["-i", castVideo],
164+ ...["-i", browserPath],
165+ "-filter_complex",
166+ [
167+ ...filters,
168+ `${acts.map((_, index) => `[act${index}]`).join("")}concat=n=${acts.length}:v=1:a=0[out]`,
169+ ].join(";"),
170+ ...["-map", "[out]"],
171+ ...["-c:v", "libx264", "-preset", "veryfast", "-crf", "24", "-movflags", "+faststart"],
172+ filmPath,
173+]);
174+rmSync(work, { recursive: true, force: true });
175+
176+// Register the film in the run's artifact list so the viewer offers it.
177+const resultPath = join(runDir, "result.json");
178+if (existsSync(resultPath)) {
179+ const result = JSON.parse(readFileSync(resultPath, "utf8")) as { artifacts?: string[] };
180+ if (Array.isArray(result.artifacts) && !result.artifacts.includes("film.mp4")) {
181+ result.artifacts.push("film.mp4");
182+ writeFileSync(resultPath, JSON.stringify(result, null, 1));
183+ }
184+}
185+
186+console.log(
187+ `film: ${filmPath}\n${acts
188+ .map(
189+ (act, index) =>
190+ ` act ${index + 1} ${act.source} ${act.from.toFixed(1)}–${act.to.toFixed(1)}s`,
191+ )
192+ .join("\n")}`,
193+);
e2e/scripts/pr-media.ts+8−4
@@ -8,8 +8,9 @@ · 7 unmodified lines
88 // the PR description.
99 //
1010 // Usage: bun e2e/scripts/pr-media.ts <run-dir-or-artifact> [...more]
11−// run dir e2e/runs/<target>/<scenario-slug> — picks session.mp4 or
12−// terminal.cast and labels the gif from result.json
11+// run dir e2e/runs/<target>/<scenario-slug> — picks film.mp4 (the
12+// whole session), else session.mp4, else terminal.cast;
13+// labels the gif from result.json
1314 // session.mp4 browser recording (ffmpeg -> gif)
1415 // terminal.cast terminal recording (agg -> gif; brew install agg)
1516 // *.png run screenshot, uploaded as-is
@@ -48,10 +49,13 @@ const resolveArtifact = (input: string): Artifact => { · 32 unmodified lines
4849 slug: basename(path).replace(/\.[^.]+$/, ""),
4950 };
5051 }
51− const recording = ["session.mp4", "terminal.cast"]
52+ // film.mp4 (scripts/film.ts) is the whole session — terminal chat AND the
53+ // browser hop, cut in time order; the bare browser session.mp4 is only the
54+ // hop, so it must never win when a film exists.
55+ const recording = ["film.mp4", "session.mp4", "terminal.cast"]
5256 .map((name) => join(path, name))
5357 .find(existsSync);
54− if (!recording) throw new Error(`${input} has no session.mp4 or terminal.cast`);
58+ if (!recording) throw new Error(`${input} has no film.mp4, session.mp4, or terminal.cast`);
5559 let label = basename(path);
5660 try {
5761 const result = JSON.parse(readFileSync(join(path, "result.json"), "utf8")) as {
e2e/src/clients/agent-chat-tui.ts+166−0
@@ -0,0 +1,166 @@
1+// The chat renderer half of the chat theater (see chat-theater.ts): a tiny
2+// agent-chat TUI that runs inside the recorded PTY and paints whatever the
3+// scenario's REAL MCP calls are doing. It performs no logic of its own — it
4+// reads base64-encoded JSON events on stdin and renders them with the
5+// pacing of a chat session (typed user input, streamed agent text, live
6+// tool spinners that run exactly as long as the real call did).
7+//
8+// Run with: bun agent-chat-tui.ts "<session title>" [events-fifo]
9+//
10+// Two transports: events arrive base64-encoded one-per-line either on stdin
11+// (PTY mode — the chat theater types them into the recorded terminal) or on
12+// a FIFO path (desk mode — the renderer runs inside a visible xterm on the
13+// virtual desktop, where stdin belongs to the terminal emulator).
14+import { createReadStream, writeFileSync } from "node:fs";
15+import { createInterface } from "node:readline";
16+
17+type TheaterEvent =
18+ | { readonly type: "user"; readonly text: string }
19+ | { readonly type: "assistant"; readonly text: string }
20+ | {
21+ readonly type: "tool-start";
22+ readonly name: string;
23+ /** The call's real input, pre-rendered as preview lines. */
24+ readonly input?: readonly string[];
25+ }
26+ | {
27+ readonly type: "tool-end";
28+ readonly ok: boolean;
29+ /** First line of the call's real result. */
30+ readonly result?: string;
31+ readonly seconds?: number;
32+ }
33+ | { readonly type: "status"; readonly text: string }
34+ | { readonly type: "done" };
35+
36+const out = (text: string) => process.stdout.write(text);
37+
38+// The PTY would otherwise echo every incoming event line into the recording.
39+if (process.stdin.isTTY) process.stdin.setRawMode(true);
40+
41+const DIM = "\x1b[2m";
42+const BOLD = "\x1b[1m";
43+const RESET = "\x1b[0m";
44+const CYAN = "\x1b[36m";
45+const GREEN = "\x1b[32m";
46+const RED = "\x1b[31m";
47+const MAGENTA = "\x1b[35m";
48+const CLEAR_LINE = "\x1b[2K\r";
49+
50+const SPINNER = ["⠋", "⠙", "⠹", "⠸", "⠼", "⠴", "⠦", "⠧", "⠇", "⠏"];
51+
52+const sleep = (ms: number) => new Promise<void>((tick) => setTimeout(tick, ms));
53+
54+const title = process.argv[2] ?? "executor session";
55+const eventsFifo = process.argv[3];
56+out(`${BOLD}${MAGENTA}●${RESET} ${BOLD}${title}${RESET}\n`);
57+out(`${DIM}${"─".repeat(Math.min(96, title.length + 24))}${RESET}\n`);
58+
59+let spinnerTimer: ReturnType<typeof setInterval> | undefined;
60+let spinnerName = "";
61+
62+// A tool call renders the way agent TUIs render them: the tool's name, the
63+// call's real input as an indented block, a spinner while the real call
64+// runs, then the real result's first line with the real duration.
65+//
66+// ⚙ execute
67+// │ const added = await tools.executor.openapi.addSpec({
68+// │ spec: { kind: "url", url: "https://…/openapi.json" },
69+// ⠼ running…
70+// ✓ 2.3s {"ok":true,"slug":"resend","toolCount":9}
71+const startTool = (name: string, input?: readonly string[]) => {
72+ spinnerName = name;
73+ out(`\n ${MAGENTA}⚙${RESET} ${BOLD}${name}${RESET}\n`);
74+ for (const line of input ?? []) {
75+ out(` ${DIM}│ ${line.slice(0, 92)}${RESET}\n`);
76+ }
77+ let frame = 0;
78+ spinnerTimer = setInterval(() => {
79+ out(`${CLEAR_LINE} ${CYAN}${SPINNER[frame % SPINNER.length]}${RESET} ${DIM}running…${RESET}`);
80+ frame += 1;
81+ }, 80);
82+};
83+
84+const endTool = (ok: boolean, result?: string, seconds?: number) => {
85+ if (spinnerTimer) clearInterval(spinnerTimer);
86+ spinnerTimer = undefined;
87+ const mark = ok ? `${GREEN}✓${RESET}` : `${RED}✗ ${spinnerName}${RESET}`;
88+ const took = seconds !== undefined ? `${DIM}${seconds.toFixed(1)}s${RESET} ` : "";
89+ out(`${CLEAR_LINE} ${mark} ${took}${DIM}${(result ?? "").slice(0, 88)}${RESET}\n`);
90+};
91+
92+const handle = async (event: TheaterEvent): Promise<boolean> => {
93+ switch (event.type) {
94+ case "user": {
95+ out(`\n${BOLD}${CYAN}┃ you${RESET} `);
96+ for (const ch of event.text) {
97+ out(ch);
98+ await sleep(18);
99+ }
100+ out("\n");
101+ return true;
102+ }
103+ case "assistant": {
104+ out(`\n${BOLD}${MAGENTA}● agent${RESET} `);
105+ for (const piece of event.text.match(/.{1,3}/gs) ?? []) {
106+ out(piece);
107+ await sleep(12);
108+ }
109+ out("\n");
110+ return true;
111+ }
112+ case "tool-start": {
113+ startTool(event.name, event.input);
114+ return true;
115+ }
116+ case "tool-end": {
117+ endTool(event.ok, event.result, event.seconds);
118+ return true;
119+ }
120+ case "status": {
121+ out(`\n ${DIM}· ${event.text}${RESET}\n`);
122+ return true;
123+ }
124+ case "done": {
125+ out(`\n${DIM}${"─".repeat(40)}${RESET}\n${GREEN}✦ session complete${RESET}\n`);
126+ if (eventsFifo) {
127+ // Desk mode: ack completion to the driver, then linger so the
128+ // closing frame stays on camera before the xterm vanishes.
129+ writeFileSync(`${eventsFifo}.done`, "");
130+ await sleep(4_000);
131+ }
132+ return false;
133+ }
134+ }
135+};
136+
137+// Events arrive faster than they render; process strictly in order so the
138+// animations (not arrival time) set the pacing.
139+const queue: TheaterEvent[] = [];
140+let draining = false;
141+const drain = async () => {
142+ if (draining) return;
143+ draining = true;
144+ while (queue.length > 0) {
145+ const event = queue.shift();
146+ if (!event) break;
147+ const keepGoing = await handle(event);
148+ if (!keepGoing) {
149+ process.exit(0);
150+ }
151+ }
152+ draining = false;
153+};
154+
155+const input = eventsFifo ? createReadStream(eventsFifo) : process.stdin;
156+createInterface({ input, terminal: false }).on("line", (line) => {
157+ const trimmed = line.trim();
158+ if (!trimmed) return;
159+ // oxlint-disable-next-line executor/no-try-catch-or-throw -- boundary: PTY stdin may echo stray keystrokes that aren't base64 JSON; ignore them
160+ try {
161+ queue.push(JSON.parse(Buffer.from(trimmed, "base64").toString("utf8")) as TheaterEvent);
162+ } catch {
163+ return;
164+ }
165+ void drain();
166+});
e2e/src/clients/chat-theater.ts+217−0
@@ -0,0 +1,217 @@
1+// Chat theater: agent-chat presentation over REAL mcporter MCP calls — no
2+// inference, no third-party agent binary. The scenario stays in Effect land
3+// making real calls; a chat renderer (agent-chat-tui.ts) paints them, so
4+// the recording reads like a developer chatting with an agent while every
5+// tool spinner brackets the genuine call it narrates.
6+//
7+// Two stages for the same play:
8+// - PTY mode (default): the renderer runs in a recorded PTY → terminal.cast
9+// - Desk mode (E2E_DESK=1): the renderer runs in a visible xterm on the
10+// virtual desktop (events over a FIFO) and the desk's single x11grab
11+// films it together with the headed browser — one screen, one mp4
12+//
13+// Division of labor in the e2e stack:
14+// - chat theater (this): deterministic PRODUCT-flow recordings
15+// - replay brain + real client (replay-brain.ts): CLIENT-behavior tests
16+// (OpenCode/Claude Code protocol handling), still no inference
17+// - real-inference evals: a separate axis entirely — performance
18+// distributions, not pass/fail scenarios
19+import { spawn, spawnSync } from "node:child_process";
20+import { createWriteStream, existsSync, mkdtempSync } from "node:fs";
21+import { tmpdir } from "node:os";
22+import { dirname, join } from "node:path";
23+import { fileURLToPath } from "node:url";
24+
25+import { Effect, Exit, Fiber } from "effect";
26+
27+import { markFocus, markRecordingStart } from "../timeline";
28+import type { CliSurface } from "../surfaces/cli";
29+
30+const RENDERER = fileURLToPath(new URL("./agent-chat-tui.ts", import.meta.url));
31+
32+interface TheaterEvent {
33+ readonly type: "user" | "assistant" | "tool-start" | "tool-end" | "status" | "done";
34+ readonly [key: string]: unknown;
35+}
36+
37+export interface ChatTheater {
38+ /** The human's chat line, typed out on screen. */
39+ readonly user: (text: string) => Effect.Effect<void>;
40+ /** The agent's reply, streamed on screen. */
41+ readonly assistant: (text: string) => Effect.Effect<void>;
42+ /** Run a REAL effect rendered as a tool call: the tool's name, the call's
43+ * real input block, a spinner for exactly as long as the call runs, then
44+ * the call's real result line with the real duration. */
45+ readonly tool: <A, E, R>(
46+ call: {
47+ /** Tool name as an agent TUI would show it (e.g. "execute"). */
48+ readonly name: string;
49+ /** The call's real input — rendered as the indented block. */
50+ readonly input?: string;
51+ /** Render the result's display line (defaults to JSON of the value). */
52+ readonly result?: (value: A) => string;
53+ },
54+ work: Effect.Effect<A, E, R>,
55+ ) => Effect.Effect<A, E, R>;
56+ /** A dim narrator line (e.g. "waiting for the browser hop"). */
57+ readonly status: (text: string) => Effect.Effect<void>;
58+}
59+
60+const sleep = (ms: number) => new Promise<void>((tick) => setTimeout(tick, ms));
61+
62+const encode = (event: TheaterEvent) =>
63+ `${Buffer.from(JSON.stringify(event)).toString("base64")}\n`;
64+
65+/** Input block, like a TUI's call preview: trimmed lines, capped count. */
66+const inputPreview = (input: string | undefined): string[] | undefined => {
67+ if (!input) return undefined;
68+ const lines = input
69+ .split("\n")
70+ .map((line) => line.trimEnd())
71+ .filter((line) => line.trim().length > 0);
72+ return lines.length > 7 ? [...lines.slice(0, 6), `… +${lines.length - 6} lines`] : lines;
73+};
74+
75+const makeTheater = (push: (event: TheaterEvent) => Effect.Effect<void>): ChatTheater => ({
76+ user: (text) => push({ type: "user", text }),
77+ assistant: (text) => push({ type: "assistant", text }),
78+ status: (text) => push({ type: "status", text }),
79+ tool: (call, work) =>
80+ Effect.gen(function* () {
81+ yield* push({ type: "tool-start", name: call.name, input: inputPreview(call.input) });
82+ const startedAt = Date.now();
83+ const exit = yield* Effect.exit(work);
84+ const seconds = (Date.now() - startedAt) / 1000;
85+ if (Exit.isSuccess(exit)) {
86+ const line = call.result ? call.result(exit.value) : JSON.stringify(exit.value);
87+ yield* push({ type: "tool-end", ok: true, result: line, seconds });
88+ return exit.value;
89+ }
90+ yield* push({ type: "tool-end", ok: false, result: "failed", seconds });
91+ return yield* exit;
92+ }),
93+});
94+
95+export interface TheaterOptions {
96+ readonly title: string;
97+ readonly record: string;
98+ readonly viewport?: { readonly cols: number; readonly rows: number };
99+}
100+
101+/**
102+ * Open the chat renderer, hand the scenario a ChatTheater handle, and close
103+ * the session (footer + exit) when the body finishes — success or failure,
104+ * so the recording always ends cleanly.
105+ */
106+export const withChatTheater = <A, E, R>(
107+ cli: CliSurface,
108+ options: TheaterOptions,
109+ body: (theater: ChatTheater) => Effect.Effect<A, E, R>,
110+): Effect.Effect<A, E, R> =>
111+ process.env.E2E_DESK === "1" ? deskTheater(options, body) : ptyTheater(cli, options, body);
112+
113+// ---------------------------------------------------------------------------
114+// PTY mode: renderer in a recorded PTY; events typed in as base64 lines.
115+// ---------------------------------------------------------------------------
116+
117+const ptyTheater = <A, E, R>(
118+ cli: CliSurface,
119+ options: TheaterOptions,
120+ body: (theater: ChatTheater) => Effect.Effect<A, E, R>,
121+): Effect.Effect<A, E, R> =>
122+ Effect.gen(function* () {
123+ const runDir = dirname(options.record);
124+ const queue: TheaterEvent[] = [];
125+ let closed = false;
126+ const push = (event: TheaterEvent) =>
127+ Effect.sync(() => {
128+ if (!closed) {
129+ // Chatting IS focusing the terminal window.
130+ if (event.type !== "done") markFocus(runDir, "terminal");
131+ queue.push(event);
132+ }
133+ });
134+
135+ const pump = cli.session(
136+ ["bun", RENDERER, options.title],
137+ async (term) => {
138+ // Don't type until the renderer has painted its header — before that
139+ // it hasn't set raw mode yet, and the PTY would echo the event line
140+ // into the recording.
141+ await term.screen.waitForText(options.title, { timeoutMs: 30_000 });
142+ // The cast's clock started with the PTY moments ago; anchor it for
143+ // the run's focus timeline (scripts/film.ts cuts on these).
144+ markRecordingStart(runDir, "terminal");
145+ const deadline = Date.now() + 30 * 60 * 1000;
146+ for (;;) {
147+ const event = queue.shift();
148+ if (event) {
149+ await term.keyboard.type(encode(event));
150+ if (event.type === "done") break;
151+ continue;
152+ }
153+ if (Date.now() > deadline) break;
154+ await sleep(40);
155+ }
156+ await term.screen.waitForText("session complete", { timeoutMs: 60_000 });
157+ },
158+ {
159+ record: options.record,
160+ viewport: options.viewport ?? { cols: 100, rows: 32 },
161+ },
162+ );
163+ const pumpFiber = yield* Effect.forkChild(Effect.exit(pump));
164+
165+ const result = yield* Effect.exit(body(makeTheater(push)));
166+ yield* push({ type: "done" });
167+ closed = true;
168+ yield* Fiber.join(pumpFiber);
169+ return yield* result;
170+ });
171+
172+// ---------------------------------------------------------------------------
173+// Desk mode: renderer in a visible xterm on the virtual desktop; events over
174+// a FIFO. No cast — the desk's x11grab films the screen.
175+// ---------------------------------------------------------------------------
176+
177+const deskTheater = <A, E, R>(
178+ options: TheaterOptions,
179+ body: (theater: ChatTheater) => Effect.Effect<A, E, R>,
180+): Effect.Effect<A, E, R> =>
181+ Effect.gen(function* () {
182+ const root = mkdtempSync(join(tmpdir(), "desk-theater-"));
183+ const fifo = join(root, "events.fifo");
184+ spawnSync("mkfifo", [fifo]);
185+
186+ const xterm = spawn(
187+ "xterm",
188+ [
189+ // -u8 + a UTF-8 locale: the renderer speaks box-drawing and bullets.
190+ ...["-u8", "-fa", "DejaVu Sans Mono", "-fs", "13"],
191+ ...["-bg", "#0b0b10", "-fg", "#e8e8ea"],
192+ ...["-geometry", "112x34+48+40", "-T", options.title],
193+ ...["-e", "bun", RENDERER, options.title, fifo],
194+ ],
195+ { stdio: "ignore", env: { ...process.env, LANG: "C.UTF-8", LC_ALL: "C.UTF-8" } },
196+ );
197+ // Opening the write end parks until the renderer opens the read end.
198+ const writer = createWriteStream(fifo);
199+ const push = (event: TheaterEvent) =>
200+ Effect.sync(() => {
201+ writer.write(encode(event));
202+ });
203+
204+ const result = yield* Effect.exit(body(makeTheater(push)));
205+ yield* push({ type: "done" });
206+ yield* Effect.promise(async () => {
207+ // Renderer acks the footer, lingers for the camera, then exits with
208+ // its xterm. Don't hang the run on a wedged terminal.
209+ const ackDeadline = Date.now() + 30_000;
210+ while (!existsSync(`${fifo}.done`) && Date.now() < ackDeadline) await sleep(200);
211+ const exitDeadline = Date.now() + 10_000;
212+ while (xterm.exitCode === null && Date.now() < exitDeadline) await sleep(200);
213+ if (xterm.exitCode === null) xterm.kill();
214+ writer.end();
215+ });
216+ return yield* result;
217+ });
e2e/src/clients/opencode.ts+28−2
@@ -24,8 +24,18 @@ export interface OpenCodeHome { · 23 unmodified lines
2424 ) => { accessToken?: string; refreshToken?: string; expiresAt?: number } | undefined;
2525 }
2626
27−/** A throwaway OpenCode installation configured with one remote MCP server. */
28−export const makeOpenCodeHome = (serverName: string, mcpUrl: string): OpenCodeHome => {
27+/** A throwaway OpenCode installation configured with one remote MCP server.
28+ *
29+ * With `chatBrainUrl` set, the config also declares a `replay` provider
30+ * pointing OpenCode's LLM traffic at a local replay brain
31+ * (clients/replay-brain.ts) and selects it as the model — real agent,
32+ * scripted conversation. Tool permissions are pre-allowed so the recorded
33+ * TUI session flows without approval dialogs. */
34+export const makeOpenCodeHome = (
35+ serverName: string,
36+ mcpUrl: string,
37+ options?: { readonly chatBrainUrl?: string },
38+): OpenCodeHome => {
2939 const root = mkdtempSync(join(tmpdir(), "e2e-opencode-"));
3040 const projectDir = join(root, "project");
3141 const dataDir = join(root, "data");
@@ -38,6 +48,22 @@ export const makeOpenCodeHome = (serverName: string, mcpUrl: string): OpenCodeHo · 6 unmodified lines
3848 JSON.stringify({
3949 $schema: "https://opencode.ai/config.json",
4050 mcp: { [serverName]: { type: "remote", url: mcpUrl } },
51+ ...(options?.chatBrainUrl
52+ ? {
53+ autoupdate: false,
54+ share: "disabled",
55+ model: "replay/replay-model",
56+ permission: { "*": "allow" },
57+ provider: {
58+ replay: {
59+ name: "Replay",
60+ npm: "@ai-sdk/openai-compatible",
61+ options: { baseURL: options.chatBrainUrl, apiKey: "replay-key" },
62+ models: { "replay-model": { name: "Replay Model" } },
63+ },
64+ },
65+ }
66+ : {}),
4167 }),
4268 );
4369 // OpenCode launches the OAuth URL via `open`; the shim records it instead.
e2e/src/clients/replay-brain.ts+208−0
@@ -0,0 +1,208 @@
1+// Replay brain: a local server speaking the OpenAI chat-completions wire
2+// (SSE streaming, tool calls) that a REAL agent (OpenCode) uses as its LLM.
3+// The scenario scripts the brain; the agent does everything else for real —
4+// TUI rendering, MCP discovery/auth, tool execution against the target,
5+// result round-trips. "Replay brain, real hands": deterministic conversation,
6+// zero modeled client behavior.
7+//
8+// The brain is a state machine, not a fixed turn list: each request gets a
9+// normalized view of the conversation so far and the scenario's `respond`
10+// callback decides what the "model" says next (text, a tool call, or both).
11+// That lets a script absorb variable-length detours — approval pauses that
12+// need a resume call, retries — without index bookkeeping.
13+import { createServer, type Server } from "node:http";
14+import type { AddressInfo } from "node:net";
15+
16+import { Effect, Scope } from "effect";
17+
18+/** What the scenario's script sees per request. */
19+export interface BrainContext {
20+ /** 0-based count of requests served before this one. */
21+ readonly requestIndex: number;
22+ /** Role of the final message — "user" means a fresh human turn, "tool"
23+ * means the agent is returning a tool result for the brain to continue. */
24+ readonly lastRole: string;
25+ /** The latest user-role message content (the human's chat input). */
26+ readonly lastUser: string;
27+ /** The latest tool-role message content (last tool result), if any. */
28+ readonly lastToolResult: string | undefined;
29+ /** Tool names offered in this request (already namespaced by the agent). */
30+ readonly toolNames: ReadonlyArray<string>;
31+}
32+
33+/** What the "model" does next. `tool.name` may be a suffix — the brain
34+ * resolves it against the request's offered tool names, so scripts don't
35+ * hardcode the agent's MCP namespacing. */
36+export interface BrainResponse {
37+ readonly text?: string;
38+ readonly tool?: { readonly name: string; readonly args: unknown };
39+}
40+
41+export interface ReplayBrain {
42+ /** Base URL for the agent's provider config (…/v1). */
43+ readonly baseUrl: string;
44+ /** Every request body served, in order (for post-hoc assertions). */
45+ readonly requests: () => ReadonlyArray<BrainRequest>;
46+ /** Script errors (mismatched expectations, throws) — assert empty. */
47+ readonly errors: () => ReadonlyArray<string>;
48+}
49+
50+export interface BrainRequest {
51+ readonly messages: ReadonlyArray<{ role: string; content: string }>;
52+ readonly toolNames: ReadonlyArray<string>;
53+}
54+
55+interface WireMessage {
56+ readonly role: string;
57+ readonly content?: unknown;
58+ readonly tool_calls?: ReadonlyArray<unknown>;
59+}
60+
61+interface WireBody {
62+ readonly messages?: ReadonlyArray<WireMessage>;
63+ readonly tools?: ReadonlyArray<{ function?: { name?: string } }>;
64+}
65+
66+const contentText = (content: unknown): string => {
67+ if (typeof content === "string") return content;
68+ if (Array.isArray(content)) {
69+ return content
70+ .map((part) =>
71+ typeof part === "object" && part !== null && "text" in part
72+ ? String((part as { text: unknown }).text)
73+ : "",
74+ )
75+ .join("");
76+ }
77+ return "";
78+};
79+
80+const sseChunk = (delta: Record<string, unknown>, finish: string | null) =>
81+ `data: ${JSON.stringify({
82+ id: "chatcmpl-replay",
83+ object: "chat.completion.chunk",
84+ created: 0,
85+ model: "replay-model",
86+ choices: [{ index: 0, delta, finish_reason: finish }],
87+ })}\n\n`;
88+
89+/**
90+ * Serve a scripted brain for the lifetime of the surrounding scope.
91+ * `respond` is called once per chat-completions request.
92+ */
93+export const serveReplayBrain = (
94+ respond: (ctx: BrainContext) => BrainResponse,
95+): Effect.Effect<ReplayBrain, never, Scope.Scope> =>
96+ Effect.acquireRelease(
97+ Effect.callback<{ server: Server; brain: ReplayBrain }>((resume) => {
98+ const served: BrainRequest[] = [];
99+ const errors: string[] = [];
100+
101+ const server = createServer((request, response) => {
102+ if (!request.url?.includes("/chat/completions")) {
103+ response.writeHead(404).end();
104+ return;
105+ }
106+ let raw = "";
107+ request.on("data", (piece: Buffer) => (raw += piece.toString("utf8")));
108+ request.on("end", () => {
109+ const body = JSON.parse(raw || "{}") as WireBody;
110+ const messages = (body.messages ?? []).map((message) => ({
111+ role: message.role,
112+ content: contentText(message.content),
113+ }));
114+ const toolNames = (body.tools ?? [])
115+ .map((tool) => tool.function?.name ?? "")
116+ .filter(Boolean);
117+ const requestIndex = served.length;
118+ served.push({ messages, toolNames });
119+
120+ const lastOf = (role: string) =>
121+ [...messages].reverse().find((message) => message.role === role)?.content;
122+
123+ let scripted: BrainResponse;
124+ // oxlint-disable-next-line executor/no-try-catch-or-throw -- boundary: a throwing script must surface as a recorded error + a stop response, not a hung agent
125+ try {
126+ scripted = respond({
127+ requestIndex,
128+ lastRole: messages.at(-1)?.role ?? "",
129+ lastUser: lastOf("user") ?? "",
130+ lastToolResult: lastOf("tool"),
131+ toolNames,
132+ });
133+ } catch (error) {
134+ errors.push(`respond() threw on request ${requestIndex}: ${String(error)}`);
135+ scripted = { text: "(replay brain script error)" };
136+ }
137+
138+ // Resolve a scripted tool name against the agent's namespaced names.
139+ let resolvedTool: { name: string; args: unknown } | undefined;
140+ if (scripted.tool) {
141+ const wanted = scripted.tool.name;
142+ const match =
143+ toolNames.find((name) => name === wanted) ??
144+ toolNames.find((name) => name.endsWith(wanted));
145+ if (match) {
146+ resolvedTool = { name: match, args: scripted.tool.args };
147+ } else {
148+ errors.push(
149+ `request ${requestIndex}: no offered tool matches "${wanted}" (offered: ${toolNames.join(", ")})`,
150+ );
151+ }
152+ }
153+
154+ response.writeHead(200, {
155+ "content-type": "text/event-stream",
156+ "cache-control": "no-cache",
157+ connection: "keep-alive",
158+ });
159+ response.write(sseChunk({ role: "assistant" }, null));
160+ if (scripted.text) {
161+ // A few chunks so the TUI streams like a real model reply.
162+ for (const piece of scripted.text.match(/.{1,24}/gs) ?? []) {
163+ response.write(sseChunk({ content: piece }, null));
164+ }
165+ }
166+ if (resolvedTool) {
167+ response.write(
168+ sseChunk(
169+ {
170+ tool_calls: [
171+ {
172+ index: 0,
173+ id: `call_${requestIndex}`,
174+ type: "function",
175+ function: {
176+ name: resolvedTool.name,
177+ arguments: JSON.stringify(resolvedTool.args ?? {}),
178+ },
179+ },
180+ ],
181+ },
182+ null,
183+ ),
184+ );
185+ }
186+ response.write(sseChunk({}, resolvedTool ? "tool_calls" : "stop"));
187+ response.write("data: [DONE]\n\n");
188+ response.end();
189+ });
190+ });
191+
192+ server.listen(0, "127.0.0.1", () => {
193+ const { port } = server.address() as AddressInfo;
194+ resume(
195+ Effect.succeed({
196+ server,
197+ brain: {
198+ baseUrl: `http://127.0.0.1:${port}/v1`,
199+ requests: () => served,
200+ errors: () => errors,
201+ },
202+ }),
203+ );
204+ });
205+ }),
206+ (resource: { server: Server; brain: ReplayBrain }) =>
207+ Effect.sync(() => void resource.server.close()),
208+ ).pipe(Effect.map(({ brain }) => brain));
e2e/src/scenario.ts+25−1
@@ -9,7 +9,8 @@ · 8 unmodified lines
99 // recording layer. What survives per run is a small result.json (for the
1010 // scenario × target matrix) plus whatever artifacts the surfaces produced
1111 // (browser video/trace/screenshots, terminal casts).
12−import { mkdirSync, readFileSync, readdirSync, rmSync, writeFileSync } from "node:fs";
12+import { execFileSync } from "node:child_process";
13+import { existsSync, mkdirSync, readFileSync, readdirSync, rmSync, writeFileSync } from "node:fs";
1314 import { join } from "node:path";
1415 import { fileURLToPath } from "node:url";
1516
@@ -136,6 +137,29 @@ export const scenario = ( · 120 unmodified lines
136137 1,
137138 ),
138139 );
140+ // A run with both recordings is ONE developer session — splice them
141+ // into film.mp4 (scripts/film.ts cuts on the focus timeline) so the
142+ // viewer plays a single recording, not parts. Best-effort: missing
143+ // agg/ffmpeg or a film failure never fails the run; the parts stay
144+ // and the viewer falls back to cast + video in story order.
145+ if (
146+ exit._tag === "Success" &&
147+ existsSync(join(dir, "terminal.cast")) &&
148+ existsSync(join(dir, "session.mp4"))
149+ ) {
150+ yield* Effect.sync(() => {
151+ // oxlint-disable-next-line executor/no-try-catch-or-throw -- boundary: optional post-processing over external tooling (agg, ffmpeg)
152+ try {
153+ execFileSync(
154+ "bun",
155+ [fileURLToPath(new URL("../scripts/film.ts", import.meta.url)), dir],
156+ { stdio: "pipe", timeout: 120_000 },
157+ );
158+ } catch {
159+ // parts remain the artifacts
160+ }
161+ });
162+ }
139163 buildManifest(RUNS_DIR);
140164 if (exit._tag === "Failure") {
141165 return yield* Effect.failCause(exit.cause);
e2e/src/surfaces/browser.ts+17−1
@@ -11,6 +11,7 @@ import { promisify } from "node:util"; · 10 unmodified lines
1111 import { Effect } from "effect";
1212 import { chromium, type Page } from "playwright";
1313
14+import { markFocus, markRecordingStart } from "../timeline";
1415 import type { Identity, Target } from "../target";
1516
1617 export interface BrowserSession {
@@ -42,7 +43,17 @@ export const makeBrowserSurface = (dir: string, target: Target): BrowserSurface · 25 unmodified lines
4243 const videoTmp = join(dir, ".video-tmp");
4344 mkdirSync(videoTmp, { recursive: true });
4445
45− const browser = await chromium.launch();
46+ // On the desk (E2E_DESK), the browser is a real headed window on the
47+ // virtual display — the desk's single screen recording films it next
48+ // to the chat terminal, exactly like a developer tabbing over.
49+ const browser = await chromium.launch(
50+ process.env.E2E_DESK === "1"
51+ ? {
52+ headless: false,
53+ args: ["--window-position=300,40", "--window-size=1100,830"],
54+ }
55+ : {},
56+ );
4657 const context = await browser.newContext({
4758 colorScheme: "dark",
4859 viewport: { width: 1280, height: 800 },
@@ -56,11 +67,16 @@ export const makeBrowserSurface = (dir: string, target: Target): BrowserSurface · 7 unmodified lines
5667 );
5768 }
5869 const page = await context.newPage();
70+ // The session video's clock starts with the page; anchor it for the
71+ // run's focus timeline (scripts/film.ts cuts on these).
72+ markRecordingStart(dir, "browser");
5973 return { browser, context, page, videoTmp, shots: { count: 0 } };
6074 }),
6175 ({ page, context, shots }) =>
6276 Effect.promise(async () => {
6377 const step = async (label: string, action: (page: Page) => Promise<void>) => {
78+ // Acting on the page IS focusing the browser window.
79+ markFocus(dir, "browser");
6480 await context.tracing.group(label);
6581 try {
6682 await action(page);
e2e/src/timeline.ts+47−0
@@ -0,0 +1,47 @@
1+// The run's focus timeline: which window the scenario was acting on, when.
2+//
3+// Focus is DERIVED, never declared — driving a Playwright page focuses the
4+// browser window; pushing a chat/terminal event focuses the terminal. The
5+// surfaces call markFocus as a side effect of normal operations, so any
6+// scenario gets a faithful "where was the developer looking" track for
7+// free, and scripts/film.ts can cut the session recordings exactly where
8+// the action moved. Anchors map wall-clock to each recording's own clock.
9+import { existsSync, readFileSync, writeFileSync } from "node:fs";
10+import { join } from "node:path";
11+
12+export type TimelineWindow = "terminal" | "browser";
13+
14+export interface Timeline {
15+ /** Wall-clock ms when each recording's clock started. */
16+ readonly anchors: { terminal?: number; browser?: number };
17+ /** Focus transitions (first event per contiguous run of a window). */
18+ readonly focus: Array<{ at: number; window: TimelineWindow }>;
19+}
20+
21+const fileFor = (runDir: string) => join(runDir, "timeline.json");
22+
23+const read = (runDir: string): Timeline => {
24+ const file = fileFor(runDir);
25+ if (!existsSync(file)) return { anchors: {}, focus: [] };
26+ return JSON.parse(readFileSync(file, "utf8")) as Timeline;
27+};
28+
29+const write = (runDir: string, timeline: Timeline) =>
30+ writeFileSync(fileFor(runDir), JSON.stringify(timeline, null, 1));
31+
32+/** Record that `window`'s recording clock starts now. */
33+export const markRecordingStart = (runDir: string, window: TimelineWindow): void => {
34+ const timeline = read(runDir);
35+ write(runDir, { ...timeline, anchors: { ...timeline.anchors, [window]: Date.now() } });
36+};
37+
38+/** Record that the scenario is acting on `window` (deduped per run). */
39+export const markFocus = (runDir: string, window: TimelineWindow): void => {
40+ const timeline = read(runDir);
41+ if (timeline.focus.at(-1)?.window === window) return;
42+ timeline.focus.push({ at: Date.now(), window });
43+ write(runDir, timeline);
44+};
45+
46+export const readTimeline = (runDir: string): Timeline | null =>
47+ existsSync(fileFor(runDir)) ? read(runDir) : null;
e2e/viewer/src/App.tsx+17−6
@@ -155,8 +155,13 @@ const RunView = ({ target, slug }: { target: string; slug: string }) => { · 154 unmodified lines
155155
156156 const has = (name: string) => result.artifacts.includes(name);
157157 const screenshots = result.artifacts.filter((a) => a.endsWith(".png")).sort();
158− const video = has("session.mp4") ? "session.mp4" : has("session.webm") ? "session.webm" : null;
159− const cast = has("terminal.cast") ? "terminal.cast" : null;
158+ // film.mp4 (scripts/film.ts) is the whole session as one video — terminal,
159+ // browser hop, terminal, cut like tabbing between full-screen windows.
160+ // When present it IS the session; the parts stay on disk for debugging.
161+ const film = has("film.mp4") ? "film.mp4" : null;
162+ const video =
163+ film ?? (has("session.mp4") ? "session.mp4" : has("session.webm") ? "session.webm" : null);
164+ const cast = film ? null : has("terminal.cast") ? "terminal.cast" : null;
160165 const media = video ?? cast;
161166 const traceUrl = has("trace.zip")
162167 ? `https://trace.playwright.dev/?trace=${encodeURIComponent(
@@ -193,7 +198,7 @@ const RunView = ({ target, slug }: { target: string; slug: string }) => { · 30 unmodified lines
193198 className={tab === "media" ? "tab active" : "tab"}
194199 onClick={() => setTab("media")}
195200 >
196− ▶ {video ? "video" : "terminal"}
201+ ▶ {cast && video ? "session" : video ? "video" : "terminal"}
197202 </button>
198203 <button
199204 className={tab === "source" ? "tab active" : "tab"}
@@ -209,18 +214,24 @@ const RunView = ({ target, slug }: { target: string; slug: string }) => { · 9 unmodified lines
209214 <TestSource url={`${base}/test.ts`} />
210215 </Suspense>
211216 )}
212− {cast && !video && tab === "media" && (
217+ {/* A run with BOTH recordings is one session in time order: the
218+ terminal chat is the spine, the browser video is the hop that
219+ happens in the middle of it. Show them in that order. */}
220+ {cast && tab === "media" && (
213221 <Suspense fallback={<p className="dim">loading recording…</p>}>
222+ {video && <h2 className="section">1 · the chat, in the terminal</h2>}
214223 <TerminalCast url={`${base}/${cast}`} />
215224 </Suspense>
216225 )}
217226 {video && tab === "media" && (
218227 <>
219− {/* muted is required for browsers to honor autoplay */}
228+ {cast && <h2 className="section">2 · the browser hop, mid-chat</h2>}
229+ {/* muted is required for browsers to honor autoplay; don't
230+ autoplay when it's the second act of a session */}
220231 <video
221232 className="hero-video"
222233 controls
223− autoPlay
234+ autoPlay={!cast}
224235 muted
225236 playsInline
226237 preload="auto"
vendor/emulate+1−1
@@ -1 +1 @@
1−Subproject commit 36638da8c104e92cb3522e0dca336618cdfd8205
1+Subproject commit 894206f09681d77f1d9d708a8d49f1c7dc72fa1f