History

Developer-session e2e: chat-theater recordings, focus timeline, one-mp4 films (#962)

* Developer-session e2e: real OpenCode chat on camera via a replay brain A scenario now runs the way a developer actually tests the product: chat with the real OpenCode TUI in a recorded PTY, the agent registers an API and drops a connect link in the chat (real MCP tool execution), a browser opens the link and pastes a real emulator-minted key while the terminal session stays open, and the chat resumes to verify the connection with a live send confirmed by the provider emulator's request ledger. - e2e/src/clients/replay-brain.ts: local OpenAI-wire chat-completions server (SSE + tool calls) scripted per scenario by transcript inspection, so approval pauses and OpenCode side requests don't derail it. Tool names resolve by suffix against whatever the agent offers. - clients/opencode.ts: makeOpenCodeHome optionally wires a 'replay' provider (openai-compatible) at the brain and pre-allows permissions. - scenarios/connect-handoff-session.test.ts: one continuous terminal.cast timeline with the browser hop forked concurrently; chat-brain.json transcript written as a run artifact even on failure. * Chat theater: agent-chat recordings over real mcporter calls, no inference The developer-session recording no longer needs a third-party agent binary or a faked LLM: the 'agent' is a chat presentation (agent-chat-tui.ts, a renderer in the recorded PTY) driven by chat-theater.ts while the scenario makes REAL mcporter MCP calls — OAuth, execute, approval pause/resume all genuine, each on-screen tool spinner bracketing the actual call it narrates. Runs anywhere bun runs (no opencode capability), ~28s on cloud. replay-brain.ts stays for the distinct tier where the third-party client's own protocol behavior is under test (OpenCode/Claude Code); real-inference evals remain a separate axis (performance distributions, not pass/fail scenarios). * Viewer: a run with both recordings plays as one session in time order Runs from chat-theater scenarios produce a terminal.cast (the chat) AND a session.mp4 (the browser hop). The run page previously picked one (video ?? cast), hiding the chat that led to the browser. Now both render under a single 'session' tab in story order — 1 · the chat, in the terminal; 2 · the browser hop, mid-chat — with autoplay reserved for video-only runs. * The Desk: film a scenario on one virtual desktop, one recording e2e/desk/run.sh builds a Linux image (Xvfb + openbox + xterm + headed Chromium + ffmpeg) and runs a scenario inside it with E2E_DESK=1: the chat renderer opens in a visible UTF-8 xterm (events over a FIFO instead of a PTY), the browser hop is a real headed window raised over it, and a single x11grab films the whole screen for the whole run — no splicing. The film replaces session.mp4 in the scenario's run dir; the scenario file itself is unchanged (chat-theater switches transports on E2E_DESK). First run builds the image and fills a node_modules volume; subsequent films of the connect-handoff session take ~2 minutes end to end. * film.ts: the session as one mp4 — full-screen cuts, no virtual desktop bun scripts/film.ts runs/<target>/<slug> turns a chat-theater run's two real recordings into one video that plays like a screen capture of a developer tabbing between full-screen windows: the cast (rendered without idle compression so cast time == video time) is cut at the browser hop — located by the scenario's narrator line, falling back to the largest output gap — and the browser recording fills the gap. Both segments are the genuine recordings; only the cut is editorial, exactly like tabbing. The film registers itself in result.json and the viewer plays film.mp4 as the session when present. Runs anywhere (agg + ffmpeg) — no Docker, no display server; the Desk remains for when actual window layering matters. * Focus is derived from operations: surfaces emit the run's focus timeline Acting on a window IS focusing it: a Playwright step marks the browser focused, a chat-theater event marks the terminal focused. Surfaces record these transitions (plus each recording's start anchor) into the run's timeline.json as a side effect of normal driving — no scenario changes, no declared focus anywhere. film.ts now builds its cut list from that timeline (any number of hops, each act mapped into its recording's own clock via the anchors), keeping the narrator-line/largest-gap heuristic only as a fallback for runs without one. The film is the operations, edited by where they happened. * Developer-session scenario adds Resend by served spec URL only The emulator serves its own OpenAPI document, so the scenario stops hand-writing a Resend subset and an explicit auth template — addSpec gets just the spec URL, exactly what an agent would send, and the platform derives base URL and the paste-a-token method from the spec itself. * Chat theater tool calls render like a real agent TUI chat.tool now takes the call itself — tool name, the REAL input (the sandbox code the scenario executes, shown as the indented preview block agent TUIs draw), and a result-line renderer — instead of a prose label. The renderer paints name → input block → live spinner → ✓ with the real duration and the real result's first line. The recording reads like an actual agent session: what was called, with what, how long it took, what came back. * pr-media prefers film.mp4 — the whole session, not just the browser hop * Auto-film runs that produce both recordings film.mp4 was a manual post-step, so any re-run left the viewer showing the parts (cast + browser video) instead of the spliced session. The runner now invokes scripts/film.ts after every successful run that has both terminal.cast and session.mp4 — best-effort, so missing agg/ffmpeg degrades back to the parts rather than failing the run. Every developer- session run's page now plays the single film by default. * Pin vendor/emulate at the OpenAPI-serving emulators Every REST emulator now serves its own OpenAPI document; the connect scenarios register integrations from those live spec URLs instead of hand-written specs.

Rhys Sullivan committed 5 months ago

Commit fe5181b1 parent f15f33d16 files · +1232 −15Browse files

Unified · Split

16 files
  1. e2e/AGENTS.md +29 −0
  2. e2e/desk/Dockerfile +27 −0
  3. e2e/desk/entry.sh +43 −0
  4. e2e/desk/run.sh +17 −0
  5. e2e/scenarios/connect-handoff-session.test.ts +189 −0
  6. e2e/scripts/film.ts +193 −0
  7. e2e/scripts/pr-media.ts +8 −4
  8. e2e/src/clients/agent-chat-tui.ts +166 −0
  9. e2e/src/clients/chat-theater.ts +217 −0
  10. e2e/src/clients/opencode.ts +28 −2
  11. e2e/src/clients/replay-brain.ts +208 −0
  12. e2e/src/scenario.ts +25 −1
  13. e2e/src/surfaces/browser.ts +17 −1
  14. e2e/src/timeline.ts +47 −0
  15. e2e/viewer/src/App.tsx +17 −6
  16. vendor/emulate +1 −1
e2e/AGENTS.md+29−0
@@ -133,3 +133,32 @@ When handing results to the user, follow the evidence contract in the root · 132 unmodified lines
133 values, not booleans — `expect(list).toContain(x)`, never133 values, not booleans — `expect(list).toContain(x)`, never
134 `expect(list.includes(x)).toBe(true)` — so failures show the data.134 `expect(list.includes(x)).toBe(true)` — so failures show the data.
135- Keep it deterministic: no sleeps; wait on conditions.135- Keep it deterministic: no sleeps; wait on conditions.
136
137## Developer-session recordings (chat theater + desk)
138
139Some scenarios are meant to be WATCHED — they show the product the way a
140developer actually uses it. Three tiers, pick deliberately:
141
1421. **Chat theater** (`src/clients/chat-theater.ts`): the default for
143 product-flow recordings. The "agent" is a chat renderer in a recorded
144 PTY; every tool spinner brackets a REAL mcporter MCP call (OAuth,
145 execute, approval resume). No inference, no third-party binary.
146 Exemplar: `scenarios/connect-handoff-session.test.ts`. Artifacts:
147 `terminal.cast` (the chat) + `session.mp4` (browser hops); the viewer
148 plays them in story order.
1492. **Replay brain + real client** (`src/clients/replay-brain.ts`): when the
150 third-party CLIENT's behavior is under test (OpenCode/Claude Code
151 protocol handling). A scripted OpenAI-wire server plays the LLM; the
152 real client does everything else. Script by transcript inspection, never
153 turn counting.
1543. **Real-inference evals**: a different axis (performance distributions,
155 not pass/fail). Not in this suite.
156
157**The Desk** (`desk/`): films a scenario on one virtual Linux desktop — the
158chat renderer in a visible xterm, the browser as a real headed window, one
159ffmpeg x11grab. The film replaces session.mp4 in the run dir; the scenario
160file is unchanged (chat-theater switches transports on `E2E_DESK=1`).
161
162```
163e2e/desk/run.sh [scenario] [project] # docker; first run builds + installs
164```
e2e/desk/Dockerfile+27−0
@@ -0,0 +1,27 @@
1# The Desk: a virtual Linux desktop that e2e scenarios are FILMED on — one
2# Xvfb display showing a real terminal window (the chat) and a real headed
3# Chromium (the browser hop), captured by a single ffmpeg x11grab into one
4# session.mp4. No splicing: the film is the screen.
5#
6# The repo is bind-mounted at /repo at runtime; node_modules and caches live
7# in named volumes (host artifacts are macOS-native and can't be reused).
8FROM node:22-bookworm
9
10RUN apt-get update && apt-get install -y --no-install-recommends \
11 xvfb openbox xterm ffmpeg xdotool x11-utils x11-xserver-utils \
12 fonts-dejavu-core fonts-noto-color-emoji \
13 curl unzip ca-certificates git \
14 && rm -rf /var/lib/apt/lists/*
15
16# Bun — the repo's package manager and script runner.
17RUN curl -fsSL https://bun.sh/install | bash -s "bun-v1.3.13" \
18 && ln -s /root/.bun/bin/bun /usr/local/bin/bun \
19 && ln -s /root/.bun/bin/bun /usr/local/bin/bunx
20
21# Chromium + system deps baked into the image, pinned to the repo's
22# playwright version so the runtime install check is a no-op.
23ENV PLAYWRIGHT_BROWSERS_PATH=/ms-playwright
24RUN npx -y playwright@1.60.0 install --with-deps chromium && rm -rf /root/.npm
25
26WORKDIR /repo
27ENTRYPOINT ["bash", "e2e/desk/entry.sh"]
e2e/desk/entry.sh+43−0
@@ -0,0 +1,43 @@
1#!/usr/bin/env bash
2# Desk entrypoint (runs inside the desk image): boot a display, roll one
3# camera on it, run the scenario with E2E_DESK=1 so its windows land on the
4# display, then drop the film into the scenario's run dir as session.mp4.
5set -euo pipefail
6
7SCENARIO="${DESK_SCENARIO:-scenarios/connect-handoff-session.test.ts}"
8PROJECT="${DESK_PROJECT:-cloud}"
9SIZE="${DESK_SIZE:-1440x900}"
10
11echo "[desk] installing dependencies (cached in a volume after first run)…"
12bun install --frozen-lockfile
13
14echo "[desk] display :99 at ${SIZE}"
15Xvfb :99 -screen 0 "${SIZE}x24" -nolisten tcp &
16export DISPLAY=:99
17sleep 0.5
18openbox &
19xsetroot -solid "#0b0b10" || true
20
21mkdir -p /desk-out
22echo "[desk] camera rolling"
23ffmpeg -loglevel error -f x11grab -framerate 24 -video_size "$SIZE" -i :99 \
24 -c:v libx264 -preset veryfast -crf 24 -pix_fmt yuv420p -y /desk-out/desk.mp4 &
25FFMPEG_PID=$!
26
27set +e
28(cd e2e && E2E_DESK=1 npx vitest run --project "$PROJECT" "$SCENARIO")
29STATUS=$?
30set -e
31
32sleep 1
33kill -INT "$FFMPEG_PID" 2>/dev/null || true
34wait "$FFMPEG_PID" 2>/dev/null || true
35
36# The desk film IS the session recording — it replaces the browser-only
37# video the surface recorded inside the run.
38RUN_DIR=$(ls -dt e2e/runs/"$PROJECT"/*/ 2>/dev/null | head -1)
39if [ -n "$RUN_DIR" ] && [ -f /desk-out/desk.mp4 ]; then
40 cp /desk-out/desk.mp4 "${RUN_DIR}session.mp4"
41 echo "[desk] film → ${RUN_DIR}session.mp4"
42fi
43exit "$STATUS"
e2e/desk/run.sh+17−0
@@ -0,0 +1,17 @@
1#!/usr/bin/env bash
2# Film a scenario on the Desk (one virtual desktop, one recording).
3# e2e/desk/run.sh [scenario-path] [project]
4# First run builds the image and fills the node_modules volume; later runs
5# reuse both.
6set -euo pipefail
7cd "$(dirname "$0")/../.."
8
9docker build -t executor-e2e-desk e2e/desk
10
11docker run --rm \
12 -v "$PWD":/repo \
13 -v executor-desk-node-modules:/repo/node_modules \
14 -v executor-desk-bun-cache:/root/.bun/install/cache \
15 -e DESK_SCENARIO="${1:-scenarios/connect-handoff-session.test.ts}" \
16 -e DESK_PROJECT="${2:-cloud}" \
17 executor-e2e-desk
e2e/scenarios/connect-handoff-session.test.ts+189−0
@@ -0,0 +1,189 @@
1// The connect handoff as a DEVELOPER SESSION — the way a human actually
2// tests this: an agent chat in a real terminal where the agent wires up the
3// API over MCP and drops a connect link, a browser hop to paste the key,
4// then back to the chat to prove the connection works with a live send.
5//
6// No inference, no third-party agent binary: the "agent" is the chat
7// theater (src/clients/chat-theater.ts) presenting REAL mcporter MCP calls
8// — OAuth, execute, approval pause/resume all genuine, every tool spinner
9// on screen bracketing the actual call it narrates. The provider on the
10// other side is real too (resend.emulators.dev); its request ledger is the
11// final evidence.
12import { randomBytes } from "node:crypto";
13import { join } from "node:path";
14
15import { expect } from "@effect/vitest";
16import { Effect } from "effect";
17
18import { scenario } from "../src/scenario";
19import { Browser, Cli, Mcp, RunDir, Target } from "../src/services";
20import { withChatTheater } from "../src/clients/chat-theater";
21import type { McpSession } from "../src/surfaces/mcp";
22
23const EMULATOR_BASE = "https://resend.emulators.dev";
24
25const unique = (prefix: string) => `${prefix}_${randomBytes(4).toString("hex")}`;
26
27// The emulator serves its own OpenAPI document (bearer auth, base URL in
28// `servers`) — adding by URL with nothing else is exactly what an agent
29// does, and the platform derives the paste-a-token auth method from the
30// spec's security scheme.
31const EMULATOR_SPEC_URL = `${EMULATOR_BASE}/openapi.json`;
32
33const addSpecCode = (slug: string) => `
34const added = await tools.executor.openapi.addSpec({
35 spec: { kind: "url", url: ${JSON.stringify(EMULATOR_SPEC_URL)} },
36 slug: ${JSON.stringify(slug)},
37});
38return added.ok ? { ok: true, slug: added.data.slug, toolCount: added.data.toolCount } : { ok: false, error: added.error };
39`;
40
41const createHandoffCode = (slug: string) => `
42const handoff = await tools.executor.coreTools.connections.createHandoff({
43 integration: ${JSON.stringify(slug)},
44 owner: "org",
45 label: "Resend",
46});
47return handoff.ok ? { ok: true, url: handoff.data.url } : { ok: false, error: handoff.error };
48`;
49
50const sendEmailCode = (slug: string, subject: string) => `
51const found = await tools.search({ namespace: ${JSON.stringify(slug)}, query: "send email", limit: 5 });
52const path = found.items[0]?.path;
53if (!path) return { ok: false, error: "no send tool found" };
54let t = tools;
55for (const seg of path.split(".")) t = t[seg];
56const sent = await t({
57 body: {
58 from: "onboarding@example.com",
59 to: "dev-session@example.com",
60 subject: ${JSON.stringify(subject)},
61 html: "<p>connect-handoff developer session</p>",
62 },
63});
64return { ok: sent.ok, path, result: sent.ok ? sent.data : sent.error };
65`;
66
67/** Run `execute`, auto-approving a paused execution (policy elicitation)
68 * once, and parse the sandbox's JSON return value. */
69const executeJson = (session: McpSession, code: string) =>
70 Effect.gen(function* () {
71 let result = yield* session.call("execute", { code });
72 if (result.text.includes("executionId:")) {
73 result = yield* session.approvePaused(result.text);
74 }
75 expect(result.ok, `execute completed (got: ${result.text.slice(0, 400)})`).toBe(true);
76 return JSON.parse(result.text) as Record<string, unknown>;
77 });
78
79const mintEmulatorApiKey = Effect.promise(async () => {
80 const response = await fetch(`${EMULATOR_BASE}/_emulate/credentials`, {
81 method: "POST",
82 headers: { "content-type": "application/json" },
83 body: JSON.stringify({ type: "api-key" }),
84 });
85 const body = (await response.json()) as { credential?: { token?: string } };
86 const token = body.credential?.token;
87 if (!token) throw new Error(`emulator credential mint failed: ${JSON.stringify(body)}`);
88 return token;
89});
90
91scenario(
92 "Connect · developer session: agent chat → handoff link → paste key → verified send",
93 { timeout: 240_000 },
94 Effect.scoped(
95 Effect.gen(function* () {
96 const target = yield* Target;
97 const mcp = yield* Mcp;
98 const browser = yield* Browser;
99 const cli = yield* Cli;
100 const runDir = yield* RunDir;
101
102 const integration = unique("resendsesh");
103 const emailSubject = unique("dev-session");
104 const apiKey = yield* mintEmulatorApiKey;
105 const identity = yield* target.newIdentity();
106 const session = mcp.session(identity);
107
108 yield* withChatTheater(
109 cli,
110 { title: "executor agent — connect Resend", record: join(runDir, "terminal.cast") },
111 (chat) =>
112 Effect.gen(function* () {
113 // Real MCP OAuth + tool discovery happens behind this call.
114 yield* chat.tool(
115 { name: "executor (mcp)", result: (tools) => `${tools.length} tools available` },
116 session.listTools(),
117 );
118
119 yield* chat.user(
120 "Add the Resend API to my executor and give me a link to connect my account",
121 );
122 yield* chat.assistant("I'll register the Resend API in your Executor now.");
123 const added = yield* chat.tool(
124 { name: "execute", input: addSpecCode(integration) },
125 executeJson(session, addSpecCode(integration)),
126 );
127 expect(added.ok, `addSpec succeeded: ${JSON.stringify(added)}`).toBe(true);
128
129 yield* chat.assistant("Registered. Creating your connect link…");
130 const handoff = yield* chat.tool(
131 { name: "execute", input: createHandoffCode(integration) },
132 executeJson(session, createHandoffCode(integration)),
133 );
134 expect(handoff.ok, `createHandoff succeeded: ${JSON.stringify(handoff)}`).toBe(true);
135 const handoffUrl = String(handoff.url);
136 expect(new URL(handoffUrl).origin, "handoff targets this deployment").toBe(
137 new URL(target.baseUrl).origin,
138 );
139
140 yield* chat.assistant(
141 `Open this link to connect your Resend account:\n\n${handoffUrl}\n\nTell me once you've pasted your API key.`,
142 );
143
144 // The browser hop — the terminal session stays open while the
145 // "user" pastes the key; the paste is the real add-account UI.
146 yield* chat.status("you, in the browser: opening the link and pasting the API key…");
147 yield* browser.session(identity, async ({ page, step }) => {
148 await step("Open the connect link from the chat", async () => {
149 await page.goto(handoffUrl, { waitUntil: "networkidle" });
150 await page
151 .getByRole("heading", { name: /Add connection/ })
152 .waitFor({ timeout: 15_000 });
153 });
154 await step("Paste the Resend API key and connect", async () => {
155 const credential = page.getByPlaceholder(/paste the value \/ token/i);
156 await credential.waitFor({ timeout: 15_000 });
157 await credential.fill(apiKey);
158 await page.getByRole("button", { name: "Add connection", exact: true }).click();
159 await page
160 .getByRole("heading", { name: /Add connection/ })
161 .waitFor({ state: "hidden", timeout: 20_000 });
162 });
163 });
164
165 yield* chat.user("Connected, now send a test email to prove it works");
166 yield* chat.assistant("Sending a test email through your new connection…");
167 const sent = yield* chat.tool(
168 { name: "execute", input: sendEmailCode(integration, emailSubject) },
169 executeJson(session, sendEmailCode(integration, emailSubject)),
170 );
171 expect(sent.ok, `email sent through the connection: ${JSON.stringify(sent)}`).toBe(
172 true,
173 );
174
175 yield* chat.assistant("Test email sent - your Resend connection works.");
176 }),
177 );
178
179 // Final evidence: the emulator's ledger saw the send from Executor.
180 const ledger = yield* Effect.promise(async () =>
181 (await fetch(`${EMULATOR_BASE}/_emulate/ledger`)).text(),
182 );
183 expect(
184 ledger.includes(emailSubject),
185 "the emulator request ledger recorded the test email",
186 ).toBe(true);
187 }),
188 ),
189);
e2e/scripts/film.ts+193−0
@@ -0,0 +1,193 @@
1// Film a run: turn a chat-theater run's real recordings into ONE mp4 that
2// plays like a screen recording of a developer tabbing between full-screen
3// windows — terminal, browser, terminal. Both segments are the genuine
4// recordings; the only editorial act is the cut, exactly like real tabbing.
5//
6// bun scripts/film.ts runs/<target>/<slug>
7//
8// The cut list comes from the run's focus timeline (src/timeline.ts):
9// surfaces mark focus as a side effect of acting — a Playwright step
10// focuses the browser, a chat event focuses the terminal — so the
11// operations themselves decide where the film cuts, for any number of
12// hops. Runs without a timeline fall back to the narrator-line / largest-
13// gap heuristic. The cast renders with NO idle compression so cast time
14// equals video time. Output: film.mp4, registered in result.json — the
15// viewer plays it as the session when present.
16import { execFileSync } from "node:child_process";
17import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
18import { tmpdir } from "node:os";
19import { join, resolve } from "node:path";
20
21import { readTimeline } from "../src/timeline";
22
23const runDir = resolve(process.argv[2] ?? "");
24const castPath = join(runDir, "terminal.cast");
25const browserPath = join(runDir, "session.mp4");
26if (!existsSync(castPath) || !existsSync(browserPath)) {
27 console.error(`film: need terminal.cast + session.mp4 in ${runDir}`);
28 process.exit(1);
29}
30
31const run = (cmd: string, args: string[]) => execFileSync(cmd, args, { stdio: "pipe" });
32const probeSeconds = (file: string): number =>
33 Number(
34 execFileSync("ffprobe", [
35 ...["-v", "quiet", "-show_entries", "format=duration"],
36 ...["-of", "csv=p=0", file],
37 ])
38 .toString()
39 .trim(),
40 );
41
42// ---------------------------------------------------------------------------
43// Build the cut list: acts of { source, from, to } in each source's clock.
44// ---------------------------------------------------------------------------
45
46interface CastEvent {
47 readonly at: number;
48 readonly text: string;
49}
50
51const events: CastEvent[] = readFileSync(castPath, "utf8")
52 .split("\n")
53 .slice(1)
54 .filter(Boolean)
55 .map((line) => JSON.parse(line) as [number, string, string])
56 .filter((event) => event[1] === "o")
57 .map((event) => ({ at: event[0], text: event[2] }));
58
59const castEnd = events.at(-1)?.at ?? 0;
60
61interface Act {
62 readonly source: "terminal" | "browser";
63 readonly from: number;
64 readonly to: number;
65}
66
67/** Acts from the focus timeline: each contiguous focus run becomes a cut of
68 * that window's recording, mapped via the recording-start anchors. */
69const actsFromTimeline = (): Act[] | null => {
70 const timeline = readTimeline(runDir);
71 if (!timeline || timeline.focus.length < 2) return null;
72 const terminalAnchor = timeline.anchors.terminal;
73 const browserAnchor = timeline.anchors.browser;
74 if (terminalAnchor === undefined || browserAnchor === undefined) return null;
75
76 const browserEnd = probeSeconds(browserPath);
77 const toMedia = (window: Act["source"], wallMs: number) =>
78 Math.max(0, (wallMs - (window === "terminal" ? terminalAnchor : browserAnchor)) / 1000);
79
80 const acts: Act[] = [];
81 for (let i = 0; i < timeline.focus.length; i += 1) {
82 const current = timeline.focus[i];
83 if (!current) continue;
84 const nextAt = timeline.focus[i + 1]?.at;
85 const from = i === 0 && current.window === "terminal" ? 0 : toMedia(current.window, current.at);
86 const mediaEnd = current.window === "terminal" ? castEnd : browserEnd;
87 const to =
88 nextAt === undefined ? mediaEnd : Math.min(toMedia(current.window, nextAt), mediaEnd);
89 if (to - from > 0.5) acts.push({ source: current.window, from, to });
90 }
91 return acts.length >= 2 ? acts : null;
92};
93
94/** Fallback for runs without a timeline: one browser hop located by the
95 * narrator line, or the largest output gap in the cast. */
96const actsFromHeuristic = (): Act[] => {
97 const findHop = (): { start: number; end: number } => {
98 const markerIndex = events.findIndex((event) => event.text.includes("in the browser"));
99 if (markerIndex !== -1) {
100 const start = events[markerIndex];
101 const after = events
102 .slice(markerIndex + 1)
103 .find((event) => event.at > (start?.at ?? 0) + 1 && event.text.trim().length > 0);
104 if (start && after) return { start: start.at + 0.8, end: after.at };
105 }
106 let best = { start: 0, end: 0 };
107 for (let i = 1; i < events.length; i += 1) {
108 const previous = events[i - 1];
109 const current = events[i];
110 if (previous && current && current.at - previous.at > best.end - best.start) {
111 best = { start: previous.at, end: current.at };
112 }
113 }
114 return best;
115 };
116 const hop = findHop();
117 if (hop.end - hop.start < 2) {
118 console.error(`film: no browser hop found in the cast (gap ${hop.end - hop.start}s)`);
119 process.exit(1);
120 }
121 return [
122 { source: "terminal", from: 0, to: hop.start },
123 { source: "browser", from: 0, to: probeSeconds(browserPath) },
124 { source: "terminal", from: hop.end, to: castEnd },
125 ];
126};
127
128const acts = actsFromTimeline() ?? actsFromHeuristic();
129
130// ---------------------------------------------------------------------------
131// Render + cut + concatenate onto one canvas — full-screen cuts, like
132// tabbing.
133// ---------------------------------------------------------------------------
134
135const work = mkdtempSync(join(tmpdir(), "e2e-film-"));
136const castGif = join(work, "cast.gif");
137const castVideo = join(work, "cast.mp4");
138
139run("agg", [
140 ...["--idle-time-limit", String(Math.ceil(castEnd) + 60)],
141 ...["--font-size", "16"],
142 castPath,
143 castGif,
144]);
145run("ffmpeg", [
146 ...["-y", "-i", castGif],
147 // agg's gif can have odd dimensions; libx264 requires even.
148 ...["-vf", "scale=ceil(iw/2)*2:ceil(ih/2)*2"],
149 ...["-pix_fmt", "yuv420p", "-r", "24", castVideo],
150]);
151
152const FIT =
153 "scale=1280:800:force_original_aspect_ratio=decrease,pad=1280:800:(ow-iw)/2:(oh-ih)/2:color=0x0b0b10,setsar=1,fps=24,format=yuv420p";
154
155const filmPath = join(runDir, "film.mp4");
156const inputIndex = { terminal: 0, browser: 1 } as const;
157const filters = acts.map(
158 (act, index) =>
159 `[${inputIndex[act.source]}:v]trim=${act.from.toFixed(2)}:${act.to.toFixed(2)},setpts=PTS-STARTPTS,${FIT}[act${index}]`,
160);
161run("ffmpeg", [
162 "-y",
163 ...["-i", castVideo],
164 ...["-i", browserPath],
165 "-filter_complex",
166 [
167 ...filters,
168 `${acts.map((_, index) => `[act${index}]`).join("")}concat=n=${acts.length}:v=1:a=0[out]`,
169 ].join(";"),
170 ...["-map", "[out]"],
171 ...["-c:v", "libx264", "-preset", "veryfast", "-crf", "24", "-movflags", "+faststart"],
172 filmPath,
173]);
174rmSync(work, { recursive: true, force: true });
175
176// Register the film in the run's artifact list so the viewer offers it.
177const resultPath = join(runDir, "result.json");
178if (existsSync(resultPath)) {
179 const result = JSON.parse(readFileSync(resultPath, "utf8")) as { artifacts?: string[] };
180 if (Array.isArray(result.artifacts) && !result.artifacts.includes("film.mp4")) {
181 result.artifacts.push("film.mp4");
182 writeFileSync(resultPath, JSON.stringify(result, null, 1));
183 }
184}
185
186console.log(
187 `film: ${filmPath}\n${acts
188 .map(
189 (act, index) =>
190 ` act ${index + 1} ${act.source} ${act.from.toFixed(1)}–${act.to.toFixed(1)}s`,
191 )
192 .join("\n")}`,
193);
e2e/scripts/pr-media.ts+8−4
@@ -8,8 +8,9 @@ · 7 unmodified lines
8// the PR description.8// the PR description.
9//9//
10// Usage: bun e2e/scripts/pr-media.ts <run-dir-or-artifact> [...more]10// Usage: bun e2e/scripts/pr-media.ts <run-dir-or-artifact> [...more]
11// run dir e2e/runs/<target>/<scenario-slug> — picks session.mp4 or11// run dir e2e/runs/<target>/<scenario-slug> — picks film.mp4 (the
12// terminal.cast and labels the gif from result.json12// whole session), else session.mp4, else terminal.cast;
13// labels the gif from result.json
13// session.mp4 browser recording (ffmpeg -> gif)14// session.mp4 browser recording (ffmpeg -> gif)
14// terminal.cast terminal recording (agg -> gif; brew install agg)15// terminal.cast terminal recording (agg -> gif; brew install agg)
15// *.png run screenshot, uploaded as-is16// *.png run screenshot, uploaded as-is
@@ -48,10 +49,13 @@ const resolveArtifact = (input: string): Artifact => { · 32 unmodified lines
48 slug: basename(path).replace(/\.[^.]+$/, ""),49 slug: basename(path).replace(/\.[^.]+$/, ""),
49 };50 };
50 }51 }
51 const recording = ["session.mp4", "terminal.cast"]52 // film.mp4 (scripts/film.ts) is the whole session — terminal chat AND the
53 // browser hop, cut in time order; the bare browser session.mp4 is only the
54 // hop, so it must never win when a film exists.
55 const recording = ["film.mp4", "session.mp4", "terminal.cast"]
52 .map((name) => join(path, name))56 .map((name) => join(path, name))
53 .find(existsSync);57 .find(existsSync);
54 if (!recording) throw new Error(`${input} has no session.mp4 or terminal.cast`);58 if (!recording) throw new Error(`${input} has no film.mp4, session.mp4, or terminal.cast`);
55 let label = basename(path);59 let label = basename(path);
56 try {60 try {
57 const result = JSON.parse(readFileSync(join(path, "result.json"), "utf8")) as {61 const result = JSON.parse(readFileSync(join(path, "result.json"), "utf8")) as {
e2e/src/clients/agent-chat-tui.ts+166−0
@@ -0,0 +1,166 @@
1// The chat renderer half of the chat theater (see chat-theater.ts): a tiny
2// agent-chat TUI that runs inside the recorded PTY and paints whatever the
3// scenario's REAL MCP calls are doing. It performs no logic of its own — it
4// reads base64-encoded JSON events on stdin and renders them with the
5// pacing of a chat session (typed user input, streamed agent text, live
6// tool spinners that run exactly as long as the real call did).
7//
8// Run with: bun agent-chat-tui.ts "<session title>" [events-fifo]
9//
10// Two transports: events arrive base64-encoded one-per-line either on stdin
11// (PTY mode — the chat theater types them into the recorded terminal) or on
12// a FIFO path (desk mode — the renderer runs inside a visible xterm on the
13// virtual desktop, where stdin belongs to the terminal emulator).
14import { createReadStream, writeFileSync } from "node:fs";
15import { createInterface } from "node:readline";
16
17type TheaterEvent =
18 | { readonly type: "user"; readonly text: string }
19 | { readonly type: "assistant"; readonly text: string }
20 | {
21 readonly type: "tool-start";
22 readonly name: string;
23 /** The call's real input, pre-rendered as preview lines. */
24 readonly input?: readonly string[];
25 }
26 | {
27 readonly type: "tool-end";
28 readonly ok: boolean;
29 /** First line of the call's real result. */
30 readonly result?: string;
31 readonly seconds?: number;
32 }
33 | { readonly type: "status"; readonly text: string }
34 | { readonly type: "done" };
35
36const out = (text: string) => process.stdout.write(text);
37
38// The PTY would otherwise echo every incoming event line into the recording.
39if (process.stdin.isTTY) process.stdin.setRawMode(true);
40
41const DIM = "\x1b[2m";
42const BOLD = "\x1b[1m";
43const RESET = "\x1b[0m";
44const CYAN = "\x1b[36m";
45const GREEN = "\x1b[32m";
46const RED = "\x1b[31m";
47const MAGENTA = "\x1b[35m";
48const CLEAR_LINE = "\x1b[2K\r";
49
50const SPINNER = ["⠋", "⠙", "⠹", "⠸", "⠼", "⠴", "⠦", "⠧", "⠇", "⠏"];
51
52const sleep = (ms: number) => new Promise<void>((tick) => setTimeout(tick, ms));
53
54const title = process.argv[2] ?? "executor session";
55const eventsFifo = process.argv[3];
56out(`${BOLD}${MAGENTA}●${RESET} ${BOLD}${title}${RESET}\n`);
57out(`${DIM}${"─".repeat(Math.min(96, title.length + 24))}${RESET}\n`);
58
59let spinnerTimer: ReturnType<typeof setInterval> | undefined;
60let spinnerName = "";
61
62// A tool call renders the way agent TUIs render them: the tool's name, the
63// call's real input as an indented block, a spinner while the real call
64// runs, then the real result's first line with the real duration.
65//
66// ⚙ execute
67// │ const added = await tools.executor.openapi.addSpec({
68// │ spec: { kind: "url", url: "https://…/openapi.json" },
69// ⠼ running…
70// ✓ 2.3s {"ok":true,"slug":"resend","toolCount":9}
71const startTool = (name: string, input?: readonly string[]) => {
72 spinnerName = name;
73 out(`\n ${MAGENTA}⚙${RESET} ${BOLD}${name}${RESET}\n`);
74 for (const line of input ?? []) {
75 out(` ${DIM}│ ${line.slice(0, 92)}${RESET}\n`);
76 }
77 let frame = 0;
78 spinnerTimer = setInterval(() => {
79 out(`${CLEAR_LINE} ${CYAN}${SPINNER[frame % SPINNER.length]}${RESET} ${DIM}running…${RESET}`);
80 frame += 1;
81 }, 80);
82};
83
84const endTool = (ok: boolean, result?: string, seconds?: number) => {
85 if (spinnerTimer) clearInterval(spinnerTimer);
86 spinnerTimer = undefined;
87 const mark = ok ? `${GREEN}✓${RESET}` : `${RED}✗ ${spinnerName}${RESET}`;
88 const took = seconds !== undefined ? `${DIM}${seconds.toFixed(1)}s${RESET} ` : "";
89 out(`${CLEAR_LINE} ${mark} ${took}${DIM}${(result ?? "").slice(0, 88)}${RESET}\n`);
90};
91
92const handle = async (event: TheaterEvent): Promise<boolean> => {
93 switch (event.type) {
94 case "user": {
95 out(`\n${BOLD}${CYAN}┃ you${RESET} `);
96 for (const ch of event.text) {
97 out(ch);
98 await sleep(18);
99 }
100 out("\n");
101 return true;
102 }
103 case "assistant": {
104 out(`\n${BOLD}${MAGENTA}● agent${RESET} `);
105 for (const piece of event.text.match(/.{1,3}/gs) ?? []) {
106 out(piece);
107 await sleep(12);
108 }
109 out("\n");
110 return true;
111 }
112 case "tool-start": {
113 startTool(event.name, event.input);
114 return true;
115 }
116 case "tool-end": {
117 endTool(event.ok, event.result, event.seconds);
118 return true;
119 }
120 case "status": {
121 out(`\n ${DIM}· ${event.text}${RESET}\n`);
122 return true;
123 }
124 case "done": {
125 out(`\n${DIM}${"─".repeat(40)}${RESET}\n${GREEN}✦ session complete${RESET}\n`);
126 if (eventsFifo) {
127 // Desk mode: ack completion to the driver, then linger so the
128 // closing frame stays on camera before the xterm vanishes.
129 writeFileSync(`${eventsFifo}.done`, "");
130 await sleep(4_000);
131 }
132 return false;
133 }
134 }
135};
136
137// Events arrive faster than they render; process strictly in order so the
138// animations (not arrival time) set the pacing.
139const queue: TheaterEvent[] = [];
140let draining = false;
141const drain = async () => {
142 if (draining) return;
143 draining = true;
144 while (queue.length > 0) {
145 const event = queue.shift();
146 if (!event) break;
147 const keepGoing = await handle(event);
148 if (!keepGoing) {
149 process.exit(0);
150 }
151 }
152 draining = false;
153};
154
155const input = eventsFifo ? createReadStream(eventsFifo) : process.stdin;
156createInterface({ input, terminal: false }).on("line", (line) => {
157 const trimmed = line.trim();
158 if (!trimmed) return;
159 // oxlint-disable-next-line executor/no-try-catch-or-throw -- boundary: PTY stdin may echo stray keystrokes that aren't base64 JSON; ignore them
160 try {
161 queue.push(JSON.parse(Buffer.from(trimmed, "base64").toString("utf8")) as TheaterEvent);
162 } catch {
163 return;
164 }
165 void drain();
166});
e2e/src/clients/chat-theater.ts+217−0
@@ -0,0 +1,217 @@
1// Chat theater: agent-chat presentation over REAL mcporter MCP calls — no
2// inference, no third-party agent binary. The scenario stays in Effect land
3// making real calls; a chat renderer (agent-chat-tui.ts) paints them, so
4// the recording reads like a developer chatting with an agent while every
5// tool spinner brackets the genuine call it narrates.
6//
7// Two stages for the same play:
8// - PTY mode (default): the renderer runs in a recorded PTY → terminal.cast
9// - Desk mode (E2E_DESK=1): the renderer runs in a visible xterm on the
10// virtual desktop (events over a FIFO) and the desk's single x11grab
11// films it together with the headed browser — one screen, one mp4
12//
13// Division of labor in the e2e stack:
14// - chat theater (this): deterministic PRODUCT-flow recordings
15// - replay brain + real client (replay-brain.ts): CLIENT-behavior tests
16// (OpenCode/Claude Code protocol handling), still no inference
17// - real-inference evals: a separate axis entirely — performance
18// distributions, not pass/fail scenarios
19import { spawn, spawnSync } from "node:child_process";
20import { createWriteStream, existsSync, mkdtempSync } from "node:fs";
21import { tmpdir } from "node:os";
22import { dirname, join } from "node:path";
23import { fileURLToPath } from "node:url";
24
25import { Effect, Exit, Fiber } from "effect";
26
27import { markFocus, markRecordingStart } from "../timeline";
28import type { CliSurface } from "../surfaces/cli";
29
30const RENDERER = fileURLToPath(new URL("./agent-chat-tui.ts", import.meta.url));
31
32interface TheaterEvent {
33 readonly type: "user" | "assistant" | "tool-start" | "tool-end" | "status" | "done";
34 readonly [key: string]: unknown;
35}
36
37export interface ChatTheater {
38 /** The human's chat line, typed out on screen. */
39 readonly user: (text: string) => Effect.Effect<void>;
40 /** The agent's reply, streamed on screen. */
41 readonly assistant: (text: string) => Effect.Effect<void>;
42 /** Run a REAL effect rendered as a tool call: the tool's name, the call's
43 * real input block, a spinner for exactly as long as the call runs, then
44 * the call's real result line with the real duration. */
45 readonly tool: <A, E, R>(
46 call: {
47 /** Tool name as an agent TUI would show it (e.g. "execute"). */
48 readonly name: string;
49 /** The call's real input — rendered as the indented block. */
50 readonly input?: string;
51 /** Render the result's display line (defaults to JSON of the value). */
52 readonly result?: (value: A) => string;
53 },
54 work: Effect.Effect<A, E, R>,
55 ) => Effect.Effect<A, E, R>;
56 /** A dim narrator line (e.g. "waiting for the browser hop"). */
57 readonly status: (text: string) => Effect.Effect<void>;
58}
59
60const sleep = (ms: number) => new Promise<void>((tick) => setTimeout(tick, ms));
61
62const encode = (event: TheaterEvent) =>
63 `${Buffer.from(JSON.stringify(event)).toString("base64")}\n`;
64
65/** Input block, like a TUI's call preview: trimmed lines, capped count. */
66const inputPreview = (input: string | undefined): string[] | undefined => {
67 if (!input) return undefined;
68 const lines = input
69 .split("\n")
70 .map((line) => line.trimEnd())
71 .filter((line) => line.trim().length > 0);
72 return lines.length > 7 ? [...lines.slice(0, 6), `… +${lines.length - 6} lines`] : lines;
73};
74
75const makeTheater = (push: (event: TheaterEvent) => Effect.Effect<void>): ChatTheater => ({
76 user: (text) => push({ type: "user", text }),
77 assistant: (text) => push({ type: "assistant", text }),
78 status: (text) => push({ type: "status", text }),
79 tool: (call, work) =>
80 Effect.gen(function* () {
81 yield* push({ type: "tool-start", name: call.name, input: inputPreview(call.input) });
82 const startedAt = Date.now();
83 const exit = yield* Effect.exit(work);
84 const seconds = (Date.now() - startedAt) / 1000;
85 if (Exit.isSuccess(exit)) {
86 const line = call.result ? call.result(exit.value) : JSON.stringify(exit.value);
87 yield* push({ type: "tool-end", ok: true, result: line, seconds });
88 return exit.value;
89 }
90 yield* push({ type: "tool-end", ok: false, result: "failed", seconds });
91 return yield* exit;
92 }),
93});
94
95export interface TheaterOptions {
96 readonly title: string;
97 readonly record: string;
98 readonly viewport?: { readonly cols: number; readonly rows: number };
99}
100
101/**
102 * Open the chat renderer, hand the scenario a ChatTheater handle, and close
103 * the session (footer + exit) when the body finishes — success or failure,
104 * so the recording always ends cleanly.
105 */
106export const withChatTheater = <A, E, R>(
107 cli: CliSurface,
108 options: TheaterOptions,
109 body: (theater: ChatTheater) => Effect.Effect<A, E, R>,
110): Effect.Effect<A, E, R> =>
111 process.env.E2E_DESK === "1" ? deskTheater(options, body) : ptyTheater(cli, options, body);
112
113// ---------------------------------------------------------------------------
114// PTY mode: renderer in a recorded PTY; events typed in as base64 lines.
115// ---------------------------------------------------------------------------
116
117const ptyTheater = <A, E, R>(
118 cli: CliSurface,
119 options: TheaterOptions,
120 body: (theater: ChatTheater) => Effect.Effect<A, E, R>,
121): Effect.Effect<A, E, R> =>
122 Effect.gen(function* () {
123 const runDir = dirname(options.record);
124 const queue: TheaterEvent[] = [];
125 let closed = false;
126 const push = (event: TheaterEvent) =>
127 Effect.sync(() => {
128 if (!closed) {
129 // Chatting IS focusing the terminal window.
130 if (event.type !== "done") markFocus(runDir, "terminal");
131 queue.push(event);
132 }
133 });
134
135 const pump = cli.session(
136 ["bun", RENDERER, options.title],
137 async (term) => {
138 // Don't type until the renderer has painted its header — before that
139 // it hasn't set raw mode yet, and the PTY would echo the event line
140 // into the recording.
141 await term.screen.waitForText(options.title, { timeoutMs: 30_000 });
142 // The cast's clock started with the PTY moments ago; anchor it for
143 // the run's focus timeline (scripts/film.ts cuts on these).
144 markRecordingStart(runDir, "terminal");
145 const deadline = Date.now() + 30 * 60 * 1000;
146 for (;;) {
147 const event = queue.shift();
148 if (event) {
149 await term.keyboard.type(encode(event));
150 if (event.type === "done") break;
151 continue;
152 }
153 if (Date.now() > deadline) break;
154 await sleep(40);
155 }
156 await term.screen.waitForText("session complete", { timeoutMs: 60_000 });
157 },
158 {
159 record: options.record,
160 viewport: options.viewport ?? { cols: 100, rows: 32 },
161 },
162 );
163 const pumpFiber = yield* Effect.forkChild(Effect.exit(pump));
164
165 const result = yield* Effect.exit(body(makeTheater(push)));
166 yield* push({ type: "done" });
167 closed = true;
168 yield* Fiber.join(pumpFiber);
169 return yield* result;
170 });
171
172// ---------------------------------------------------------------------------
173// Desk mode: renderer in a visible xterm on the virtual desktop; events over
174// a FIFO. No cast — the desk's x11grab films the screen.
175// ---------------------------------------------------------------------------
176
177const deskTheater = <A, E, R>(
178 options: TheaterOptions,
179 body: (theater: ChatTheater) => Effect.Effect<A, E, R>,
180): Effect.Effect<A, E, R> =>
181 Effect.gen(function* () {
182 const root = mkdtempSync(join(tmpdir(), "desk-theater-"));
183 const fifo = join(root, "events.fifo");
184 spawnSync("mkfifo", [fifo]);
185
186 const xterm = spawn(
187 "xterm",
188 [
189 // -u8 + a UTF-8 locale: the renderer speaks box-drawing and bullets.
190 ...["-u8", "-fa", "DejaVu Sans Mono", "-fs", "13"],
191 ...["-bg", "#0b0b10", "-fg", "#e8e8ea"],
192 ...["-geometry", "112x34+48+40", "-T", options.title],
193 ...["-e", "bun", RENDERER, options.title, fifo],
194 ],
195 { stdio: "ignore", env: { ...process.env, LANG: "C.UTF-8", LC_ALL: "C.UTF-8" } },
196 );
197 // Opening the write end parks until the renderer opens the read end.
198 const writer = createWriteStream(fifo);
199 const push = (event: TheaterEvent) =>
200 Effect.sync(() => {
201 writer.write(encode(event));
202 });
203
204 const result = yield* Effect.exit(body(makeTheater(push)));
205 yield* push({ type: "done" });
206 yield* Effect.promise(async () => {
207 // Renderer acks the footer, lingers for the camera, then exits with
208 // its xterm. Don't hang the run on a wedged terminal.
209 const ackDeadline = Date.now() + 30_000;
210 while (!existsSync(`${fifo}.done`) && Date.now() < ackDeadline) await sleep(200);
211 const exitDeadline = Date.now() + 10_000;
212 while (xterm.exitCode === null && Date.now() < exitDeadline) await sleep(200);
213 if (xterm.exitCode === null) xterm.kill();
214 writer.end();
215 });
216 return yield* result;
217 });
e2e/src/clients/opencode.ts+28−2
@@ -24,8 +24,18 @@ export interface OpenCodeHome { · 23 unmodified lines
24 ) => { accessToken?: string; refreshToken?: string; expiresAt?: number } | undefined;24 ) => { accessToken?: string; refreshToken?: string; expiresAt?: number } | undefined;
25}25}
26 26
27/** A throwaway OpenCode installation configured with one remote MCP server. */27/** A throwaway OpenCode installation configured with one remote MCP server.
28export const makeOpenCodeHome = (serverName: string, mcpUrl: string): OpenCodeHome => {28 *
29 * With `chatBrainUrl` set, the config also declares a `replay` provider
30 * pointing OpenCode's LLM traffic at a local replay brain
31 * (clients/replay-brain.ts) and selects it as the model — real agent,
32 * scripted conversation. Tool permissions are pre-allowed so the recorded
33 * TUI session flows without approval dialogs. */
34export const makeOpenCodeHome = (
35 serverName: string,
36 mcpUrl: string,
37 options?: { readonly chatBrainUrl?: string },
38): OpenCodeHome => {
29 const root = mkdtempSync(join(tmpdir(), "e2e-opencode-"));39 const root = mkdtempSync(join(tmpdir(), "e2e-opencode-"));
30 const projectDir = join(root, "project");40 const projectDir = join(root, "project");
31 const dataDir = join(root, "data");41 const dataDir = join(root, "data");
@@ -38,6 +48,22 @@ export const makeOpenCodeHome = (serverName: string, mcpUrl: string): OpenCodeHo · 6 unmodified lines
38 JSON.stringify({48 JSON.stringify({
39 $schema: "https://opencode.ai/config.json",49 $schema: "https://opencode.ai/config.json",
40 mcp: { [serverName]: { type: "remote", url: mcpUrl } },50 mcp: { [serverName]: { type: "remote", url: mcpUrl } },
51 ...(options?.chatBrainUrl
52 ? {
53 autoupdate: false,
54 share: "disabled",
55 model: "replay/replay-model",
56 permission: { "*": "allow" },
57 provider: {
58 replay: {
59 name: "Replay",
60 npm: "@ai-sdk/openai-compatible",
61 options: { baseURL: options.chatBrainUrl, apiKey: "replay-key" },
62 models: { "replay-model": { name: "Replay Model" } },
63 },
64 },
65 }
66 : {}),
41 }),67 }),
42 );68 );
43 // OpenCode launches the OAuth URL via `open`; the shim records it instead.69 // OpenCode launches the OAuth URL via `open`; the shim records it instead.
e2e/src/clients/replay-brain.ts+208−0
@@ -0,0 +1,208 @@
1// Replay brain: a local server speaking the OpenAI chat-completions wire
2// (SSE streaming, tool calls) that a REAL agent (OpenCode) uses as its LLM.
3// The scenario scripts the brain; the agent does everything else for real —
4// TUI rendering, MCP discovery/auth, tool execution against the target,
5// result round-trips. "Replay brain, real hands": deterministic conversation,
6// zero modeled client behavior.
7//
8// The brain is a state machine, not a fixed turn list: each request gets a
9// normalized view of the conversation so far and the scenario's `respond`
10// callback decides what the "model" says next (text, a tool call, or both).
11// That lets a script absorb variable-length detours — approval pauses that
12// need a resume call, retries — without index bookkeeping.
13import { createServer, type Server } from "node:http";
14import type { AddressInfo } from "node:net";
15
16import { Effect, Scope } from "effect";
17
18/** What the scenario's script sees per request. */
19export interface BrainContext {
20 /** 0-based count of requests served before this one. */
21 readonly requestIndex: number;
22 /** Role of the final message — "user" means a fresh human turn, "tool"
23 * means the agent is returning a tool result for the brain to continue. */
24 readonly lastRole: string;
25 /** The latest user-role message content (the human's chat input). */
26 readonly lastUser: string;
27 /** The latest tool-role message content (last tool result), if any. */
28 readonly lastToolResult: string | undefined;
29 /** Tool names offered in this request (already namespaced by the agent). */
30 readonly toolNames: ReadonlyArray<string>;
31}
32
33/** What the "model" does next. `tool.name` may be a suffix — the brain
34 * resolves it against the request's offered tool names, so scripts don't
35 * hardcode the agent's MCP namespacing. */
36export interface BrainResponse {
37 readonly text?: string;
38 readonly tool?: { readonly name: string; readonly args: unknown };
39}
40
41export interface ReplayBrain {
42 /** Base URL for the agent's provider config (…/v1). */
43 readonly baseUrl: string;
44 /** Every request body served, in order (for post-hoc assertions). */
45 readonly requests: () => ReadonlyArray<BrainRequest>;
46 /** Script errors (mismatched expectations, throws) — assert empty. */
47 readonly errors: () => ReadonlyArray<string>;
48}
49
50export interface BrainRequest {
51 readonly messages: ReadonlyArray<{ role: string; content: string }>;
52 readonly toolNames: ReadonlyArray<string>;
53}
54
55interface WireMessage {
56 readonly role: string;
57 readonly content?: unknown;
58 readonly tool_calls?: ReadonlyArray<unknown>;
59}
60
61interface WireBody {
62 readonly messages?: ReadonlyArray<WireMessage>;
63 readonly tools?: ReadonlyArray<{ function?: { name?: string } }>;
64}
65
66const contentText = (content: unknown): string => {
67 if (typeof content === "string") return content;
68 if (Array.isArray(content)) {
69 return content
70 .map((part) =>
71 typeof part === "object" && part !== null && "text" in part
72 ? String((part as { text: unknown }).text)
73 : "",
74 )
75 .join("");
76 }
77 return "";
78};
79
80const sseChunk = (delta: Record<string, unknown>, finish: string | null) =>
81 `data: ${JSON.stringify({
82 id: "chatcmpl-replay",
83 object: "chat.completion.chunk",
84 created: 0,
85 model: "replay-model",
86 choices: [{ index: 0, delta, finish_reason: finish }],
87 })}\n\n`;
88
89/**
90 * Serve a scripted brain for the lifetime of the surrounding scope.
91 * `respond` is called once per chat-completions request.
92 */
93export const serveReplayBrain = (
94 respond: (ctx: BrainContext) => BrainResponse,
95): Effect.Effect<ReplayBrain, never, Scope.Scope> =>
96 Effect.acquireRelease(
97 Effect.callback<{ server: Server; brain: ReplayBrain }>((resume) => {
98 const served: BrainRequest[] = [];
99 const errors: string[] = [];
100
101 const server = createServer((request, response) => {
102 if (!request.url?.includes("/chat/completions")) {
103 response.writeHead(404).end();
104 return;
105 }
106 let raw = "";
107 request.on("data", (piece: Buffer) => (raw += piece.toString("utf8")));
108 request.on("end", () => {
109 const body = JSON.parse(raw || "{}") as WireBody;
110 const messages = (body.messages ?? []).map((message) => ({
111 role: message.role,
112 content: contentText(message.content),
113 }));
114 const toolNames = (body.tools ?? [])
115 .map((tool) => tool.function?.name ?? "")
116 .filter(Boolean);
117 const requestIndex = served.length;
118 served.push({ messages, toolNames });
119
120 const lastOf = (role: string) =>
121 [...messages].reverse().find((message) => message.role === role)?.content;
122
123 let scripted: BrainResponse;
124 // oxlint-disable-next-line executor/no-try-catch-or-throw -- boundary: a throwing script must surface as a recorded error + a stop response, not a hung agent
125 try {
126 scripted = respond({
127 requestIndex,
128 lastRole: messages.at(-1)?.role ?? "",
129 lastUser: lastOf("user") ?? "",
130 lastToolResult: lastOf("tool"),
131 toolNames,
132 });
133 } catch (error) {
134 errors.push(`respond() threw on request ${requestIndex}: ${String(error)}`);
135 scripted = { text: "(replay brain script error)" };
136 }
137
138 // Resolve a scripted tool name against the agent's namespaced names.
139 let resolvedTool: { name: string; args: unknown } | undefined;
140 if (scripted.tool) {
141 const wanted = scripted.tool.name;
142 const match =
143 toolNames.find((name) => name === wanted) ??
144 toolNames.find((name) => name.endsWith(wanted));
145 if (match) {
146 resolvedTool = { name: match, args: scripted.tool.args };
147 } else {
148 errors.push(
149 `request ${requestIndex}: no offered tool matches "${wanted}" (offered: ${toolNames.join(", ")})`,
150 );
151 }
152 }
153
154 response.writeHead(200, {
155 "content-type": "text/event-stream",
156 "cache-control": "no-cache",
157 connection: "keep-alive",
158 });
159 response.write(sseChunk({ role: "assistant" }, null));
160 if (scripted.text) {
161 // A few chunks so the TUI streams like a real model reply.
162 for (const piece of scripted.text.match(/.{1,24}/gs) ?? []) {
163 response.write(sseChunk({ content: piece }, null));
164 }
165 }
166 if (resolvedTool) {
167 response.write(
168 sseChunk(
169 {
170 tool_calls: [
171 {
172 index: 0,
173 id: `call_${requestIndex}`,
174 type: "function",
175 function: {
176 name: resolvedTool.name,
177 arguments: JSON.stringify(resolvedTool.args ?? {}),
178 },
179 },
180 ],
181 },
182 null,
183 ),
184 );
185 }
186 response.write(sseChunk({}, resolvedTool ? "tool_calls" : "stop"));
187 response.write("data: [DONE]\n\n");
188 response.end();
189 });
190 });
191
192 server.listen(0, "127.0.0.1", () => {
193 const { port } = server.address() as AddressInfo;
194 resume(
195 Effect.succeed({
196 server,
197 brain: {
198 baseUrl: `http://127.0.0.1:${port}/v1`,
199 requests: () => served,
200 errors: () => errors,
201 },
202 }),
203 );
204 });
205 }),
206 (resource: { server: Server; brain: ReplayBrain }) =>
207 Effect.sync(() => void resource.server.close()),
208 ).pipe(Effect.map(({ brain }) => brain));
e2e/src/scenario.ts+25−1
@@ -9,7 +9,8 @@ · 8 unmodified lines
9// recording layer. What survives per run is a small result.json (for the9// recording layer. What survives per run is a small result.json (for the
10// scenario × target matrix) plus whatever artifacts the surfaces produced10// scenario × target matrix) plus whatever artifacts the surfaces produced
11// (browser video/trace/screenshots, terminal casts).11// (browser video/trace/screenshots, terminal casts).
12import { mkdirSync, readFileSync, readdirSync, rmSync, writeFileSync } from "node:fs";12import { execFileSync } from "node:child_process";
13import { existsSync, mkdirSync, readFileSync, readdirSync, rmSync, writeFileSync } from "node:fs";
13import { join } from "node:path";14import { join } from "node:path";
14import { fileURLToPath } from "node:url";15import { fileURLToPath } from "node:url";
15 16
@@ -136,6 +137,29 @@ export const scenario = ( · 120 unmodified lines
136 1,137 1,
137 ),138 ),
138 );139 );
140 // A run with both recordings is ONE developer session — splice them
141 // into film.mp4 (scripts/film.ts cuts on the focus timeline) so the
142 // viewer plays a single recording, not parts. Best-effort: missing
143 // agg/ffmpeg or a film failure never fails the run; the parts stay
144 // and the viewer falls back to cast + video in story order.
145 if (
146 exit._tag === "Success" &&
147 existsSync(join(dir, "terminal.cast")) &&
148 existsSync(join(dir, "session.mp4"))
149 ) {
150 yield* Effect.sync(() => {
151 // oxlint-disable-next-line executor/no-try-catch-or-throw -- boundary: optional post-processing over external tooling (agg, ffmpeg)
152 try {
153 execFileSync(
154 "bun",
155 [fileURLToPath(new URL("../scripts/film.ts", import.meta.url)), dir],
156 { stdio: "pipe", timeout: 120_000 },
157 );
158 } catch {
159 // parts remain the artifacts
160 }
161 });
162 }
139 buildManifest(RUNS_DIR);163 buildManifest(RUNS_DIR);
140 if (exit._tag === "Failure") {164 if (exit._tag === "Failure") {
141 return yield* Effect.failCause(exit.cause);165 return yield* Effect.failCause(exit.cause);
e2e/src/surfaces/browser.ts+17−1
@@ -11,6 +11,7 @@ import { promisify } from "node:util"; · 10 unmodified lines
11import { Effect } from "effect";11import { Effect } from "effect";
12import { chromium, type Page } from "playwright";12import { chromium, type Page } from "playwright";
13 13
14import { markFocus, markRecordingStart } from "../timeline";
14import type { Identity, Target } from "../target";15import type { Identity, Target } from "../target";
15 16
16export interface BrowserSession {17export interface BrowserSession {
@@ -42,7 +43,17 @@ export const makeBrowserSurface = (dir: string, target: Target): BrowserSurface · 25 unmodified lines
42 const videoTmp = join(dir, ".video-tmp");43 const videoTmp = join(dir, ".video-tmp");
43 mkdirSync(videoTmp, { recursive: true });44 mkdirSync(videoTmp, { recursive: true });
44 45
45 const browser = await chromium.launch();46 // On the desk (E2E_DESK), the browser is a real headed window on the
47 // virtual display — the desk's single screen recording films it next
48 // to the chat terminal, exactly like a developer tabbing over.
49 const browser = await chromium.launch(
50 process.env.E2E_DESK === "1"
51 ? {
52 headless: false,
53 args: ["--window-position=300,40", "--window-size=1100,830"],
54 }
55 : {},
56 );
46 const context = await browser.newContext({57 const context = await browser.newContext({
47 colorScheme: "dark",58 colorScheme: "dark",
48 viewport: { width: 1280, height: 800 },59 viewport: { width: 1280, height: 800 },
@@ -56,11 +67,16 @@ export const makeBrowserSurface = (dir: string, target: Target): BrowserSurface · 7 unmodified lines
56 );67 );
57 }68 }
58 const page = await context.newPage();69 const page = await context.newPage();
70 // The session video's clock starts with the page; anchor it for the
71 // run's focus timeline (scripts/film.ts cuts on these).
72 markRecordingStart(dir, "browser");
59 return { browser, context, page, videoTmp, shots: { count: 0 } };73 return { browser, context, page, videoTmp, shots: { count: 0 } };
60 }),74 }),
61 ({ page, context, shots }) =>75 ({ page, context, shots }) =>
62 Effect.promise(async () => {76 Effect.promise(async () => {
63 const step = async (label: string, action: (page: Page) => Promise<void>) => {77 const step = async (label: string, action: (page: Page) => Promise<void>) => {
78 // Acting on the page IS focusing the browser window.
79 markFocus(dir, "browser");
64 await context.tracing.group(label);80 await context.tracing.group(label);
65 try {81 try {
66 await action(page);82 await action(page);
e2e/src/timeline.ts+47−0
@@ -0,0 +1,47 @@
1// The run's focus timeline: which window the scenario was acting on, when.
2//
3// Focus is DERIVED, never declared — driving a Playwright page focuses the
4// browser window; pushing a chat/terminal event focuses the terminal. The
5// surfaces call markFocus as a side effect of normal operations, so any
6// scenario gets a faithful "where was the developer looking" track for
7// free, and scripts/film.ts can cut the session recordings exactly where
8// the action moved. Anchors map wall-clock to each recording's own clock.
9import { existsSync, readFileSync, writeFileSync } from "node:fs";
10import { join } from "node:path";
11
12export type TimelineWindow = "terminal" | "browser";
13
14export interface Timeline {
15 /** Wall-clock ms when each recording's clock started. */
16 readonly anchors: { terminal?: number; browser?: number };
17 /** Focus transitions (first event per contiguous run of a window). */
18 readonly focus: Array<{ at: number; window: TimelineWindow }>;
19}
20
21const fileFor = (runDir: string) => join(runDir, "timeline.json");
22
23const read = (runDir: string): Timeline => {
24 const file = fileFor(runDir);
25 if (!existsSync(file)) return { anchors: {}, focus: [] };
26 return JSON.parse(readFileSync(file, "utf8")) as Timeline;
27};
28
29const write = (runDir: string, timeline: Timeline) =>
30 writeFileSync(fileFor(runDir), JSON.stringify(timeline, null, 1));
31
32/** Record that `window`'s recording clock starts now. */
33export const markRecordingStart = (runDir: string, window: TimelineWindow): void => {
34 const timeline = read(runDir);
35 write(runDir, { ...timeline, anchors: { ...timeline.anchors, [window]: Date.now() } });
36};
37
38/** Record that the scenario is acting on `window` (deduped per run). */
39export const markFocus = (runDir: string, window: TimelineWindow): void => {
40 const timeline = read(runDir);
41 if (timeline.focus.at(-1)?.window === window) return;
42 timeline.focus.push({ at: Date.now(), window });
43 write(runDir, timeline);
44};
45
46export const readTimeline = (runDir: string): Timeline | null =>
47 existsSync(fileFor(runDir)) ? read(runDir) : null;
e2e/viewer/src/App.tsx+17−6
@@ -155,8 +155,13 @@ const RunView = ({ target, slug }: { target: string; slug: string }) => { · 154 unmodified lines
155 155
156 const has = (name: string) => result.artifacts.includes(name);156 const has = (name: string) => result.artifacts.includes(name);
157 const screenshots = result.artifacts.filter((a) => a.endsWith(".png")).sort();157 const screenshots = result.artifacts.filter((a) => a.endsWith(".png")).sort();
158 const video = has("session.mp4") ? "session.mp4" : has("session.webm") ? "session.webm" : null;158 // film.mp4 (scripts/film.ts) is the whole session as one video — terminal,
159 const cast = has("terminal.cast") ? "terminal.cast" : null;159 // browser hop, terminal, cut like tabbing between full-screen windows.
160 // When present it IS the session; the parts stay on disk for debugging.
161 const film = has("film.mp4") ? "film.mp4" : null;
162 const video =
163 film ?? (has("session.mp4") ? "session.mp4" : has("session.webm") ? "session.webm" : null);
164 const cast = film ? null : has("terminal.cast") ? "terminal.cast" : null;
160 const media = video ?? cast;165 const media = video ?? cast;
161 const traceUrl = has("trace.zip")166 const traceUrl = has("trace.zip")
162 ? `https://trace.playwright.dev/?trace=${encodeURIComponent(167 ? `https://trace.playwright.dev/?trace=${encodeURIComponent(
@@ -193,7 +198,7 @@ const RunView = ({ target, slug }: { target: string; slug: string }) => { · 30 unmodified lines
193 className={tab === "media" ? "tab active" : "tab"}198 className={tab === "media" ? "tab active" : "tab"}
194 onClick={() => setTab("media")}199 onClick={() => setTab("media")}
195 >200 >
196 ▶ {video ? "video" : "terminal"}201 ▶ {cast && video ? "session" : video ? "video" : "terminal"}
197 </button>202 </button>
198 <button203 <button
199 className={tab === "source" ? "tab active" : "tab"}204 className={tab === "source" ? "tab active" : "tab"}
@@ -209,18 +214,24 @@ const RunView = ({ target, slug }: { target: string; slug: string }) => { · 9 unmodified lines
209 <TestSource url={`${base}/test.ts`} />214 <TestSource url={`${base}/test.ts`} />
210 </Suspense>215 </Suspense>
211 )}216 )}
212 {cast && !video && tab === "media" && (217 {/* A run with BOTH recordings is one session in time order: the
218 terminal chat is the spine, the browser video is the hop that
219 happens in the middle of it. Show them in that order. */}
220 {cast && tab === "media" && (
213 <Suspense fallback={<p className="dim">loading recording…</p>}>221 <Suspense fallback={<p className="dim">loading recording…</p>}>
222 {video && <h2 className="section">1 · the chat, in the terminal</h2>}
214 <TerminalCast url={`${base}/${cast}`} />223 <TerminalCast url={`${base}/${cast}`} />
215 </Suspense>224 </Suspense>
216 )}225 )}
217 {video && tab === "media" && (226 {video && tab === "media" && (
218 <>227 <>
219 {/* muted is required for browsers to honor autoplay */}228 {cast && <h2 className="section">2 · the browser hop, mid-chat</h2>}
229 {/* muted is required for browsers to honor autoplay; don't
230 autoplay when it's the second act of a session */}
220 <video231 <video
221 className="hero-video"232 className="hero-video"
222 controls233 controls
223 autoPlay234 autoPlay={!cast}
224 muted235 muted
225 playsInline236 playsInline
226 preload="auto"237 preload="auto"
vendor/emulate+1−1
@@ -1 +1 @@
1Subproject commit 36638da8c104e92cb3522e0dca336618cdfd82051Subproject commit 894206f09681d77f1d9d708a8d49f1c7dc72fa1f