* Developer-session e2e: real OpenCode chat on camera via a replay brain A scenario now runs the way a developer actually tests the product: chat with the real OpenCode TUI in a recorded PTY, the agent registers an API and drops a connect link in the chat (real MCP tool execution), a browser opens the link and pastes a real emulator-minted key while the terminal session stays open, and the chat resumes to verify the connection with a live send confirmed by the provider emulator's request ledger. - e2e/src/clients/replay-brain.ts: local OpenAI-wire chat-completions server (SSE + tool calls) scripted per scenario by transcript inspection, so approval pauses and OpenCode side requests don't derail it. Tool names resolve by suffix against whatever the agent offers. - clients/opencode.ts: makeOpenCodeHome optionally wires a 'replay' provider (openai-compatible) at the brain and pre-allows permissions. - scenarios/connect-handoff-session.test.ts: one continuous terminal.cast timeline with the browser hop forked concurrently; chat-brain.json transcript written as a run artifact even on failure. * Chat theater: agent-chat recordings over real mcporter calls, no inference The developer-session recording no longer needs a third-party agent binary or a faked LLM: the 'agent' is a chat presentation (agent-chat-tui.ts, a renderer in the recorded PTY) driven by chat-theater.ts while the scenario makes REAL mcporter MCP calls — OAuth, execute, approval pause/resume all genuine, each on-screen tool spinner bracketing the actual call it narrates. Runs anywhere bun runs (no opencode capability), ~28s on cloud. replay-brain.ts stays for the distinct tier where the third-party client's own protocol behavior is under test (OpenCode/Claude Code); real-inference evals remain a separate axis (performance distributions, not pass/fail scenarios). * Viewer: a run with both recordings plays as one session in time order Runs from chat-theater scenarios produce a terminal.cast (the chat) AND a session.mp4 (the browser hop). The run page previously picked one (video ?? cast), hiding the chat that led to the browser. Now both render under a single 'session' tab in story order — 1 · the chat, in the terminal; 2 · the browser hop, mid-chat — with autoplay reserved for video-only runs. * The Desk: film a scenario on one virtual desktop, one recording e2e/desk/run.sh builds a Linux image (Xvfb + openbox + xterm + headed Chromium + ffmpeg) and runs a scenario inside it with E2E_DESK=1: the chat renderer opens in a visible UTF-8 xterm (events over a FIFO instead of a PTY), the browser hop is a real headed window raised over it, and a single x11grab films the whole screen for the whole run — no splicing. The film replaces session.mp4 in the scenario's run dir; the scenario file itself is unchanged (chat-theater switches transports on E2E_DESK). First run builds the image and fills a node_modules volume; subsequent films of the connect-handoff session take ~2 minutes end to end. * film.ts: the session as one mp4 — full-screen cuts, no virtual desktop bun scripts/film.ts runs/<target>/<slug> turns a chat-theater run's two real recordings into one video that plays like a screen capture of a developer tabbing between full-screen windows: the cast (rendered without idle compression so cast time == video time) is cut at the browser hop — located by the scenario's narrator line, falling back to the largest output gap — and the browser recording fills the gap. Both segments are the genuine recordings; only the cut is editorial, exactly like tabbing. The film registers itself in result.json and the viewer plays film.mp4 as the session when present. Runs anywhere (agg + ffmpeg) — no Docker, no display server; the Desk remains for when actual window layering matters. * Focus is derived from operations: surfaces emit the run's focus timeline Acting on a window IS focusing it: a Playwright step marks the browser focused, a chat-theater event marks the terminal focused. Surfaces record these transitions (plus each recording's start anchor) into the run's timeline.json as a side effect of normal driving — no scenario changes, no declared focus anywhere. film.ts now builds its cut list from that timeline (any number of hops, each act mapped into its recording's own clock via the anchors), keeping the narrator-line/largest-gap heuristic only as a fallback for runs without one. The film is the operations, edited by where they happened. * Developer-session scenario adds Resend by served spec URL only The emulator serves its own OpenAPI document, so the scenario stops hand-writing a Resend subset and an explicit auth template — addSpec gets just the spec URL, exactly what an agent would send, and the platform derives base URL and the paste-a-token method from the spec itself. * Chat theater tool calls render like a real agent TUI chat.tool now takes the call itself — tool name, the REAL input (the sandbox code the scenario executes, shown as the indented preview block agent TUIs draw), and a result-line renderer — instead of a prose label. The renderer paints name → input block → live spinner → ✓ with the real duration and the real result's first line. The recording reads like an actual agent session: what was called, with what, how long it took, what came back. * pr-media prefers film.mp4 — the whole session, not just the browser hop * Auto-film runs that produce both recordings film.mp4 was a manual post-step, so any re-run left the viewer showing the parts (cast + browser video) instead of the spliced session. The runner now invokes scripts/film.ts after every successful run that has both terminal.cast and session.mp4 — best-effort, so missing agg/ffmpeg degrades back to the parts rather than failing the run. Every developer- session run's page now plays the single film by default. * Pin vendor/emulate at the OpenAPI-serving emulators Every REST emulator now serves its own OpenAPI document; the connect scenarios register integrations from those live spec URLs instead of hand-written specs.
Showing the changes to e2e/scripts (2 of 16 files). Show all
Unified · Split
| @@ -0,0 +1,193 @@ | ||
| 1 | +// Film a run: turn a chat-theater run's real recordings into ONE mp4 that | |
| 2 | +// plays like a screen recording of a developer tabbing between full-screen | |
| 3 | +// windows — terminal, browser, terminal. Both segments are the genuine | |
| 4 | +// recordings; the only editorial act is the cut, exactly like real tabbing. | |
| 5 | +// | |
| 6 | +// bun scripts/film.ts runs/<target>/<slug> | |
| 7 | +// | |
| 8 | +// The cut list comes from the run's focus timeline (src/timeline.ts): | |
| 9 | +// surfaces mark focus as a side effect of acting — a Playwright step | |
| 10 | +// focuses the browser, a chat event focuses the terminal — so the | |
| 11 | +// operations themselves decide where the film cuts, for any number of | |
| 12 | +// hops. Runs without a timeline fall back to the narrator-line / largest- | |
| 13 | +// gap heuristic. The cast renders with NO idle compression so cast time | |
| 14 | +// equals video time. Output: film.mp4, registered in result.json — the | |
| 15 | +// viewer plays it as the session when present. | |
| 16 | +import { execFileSync } from "node:child_process"; | |
| 17 | +import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; | |
| 18 | +import { tmpdir } from "node:os"; | |
| 19 | +import { join, resolve } from "node:path"; | |
| 20 | + | |
| 21 | +import { readTimeline } from "../src/timeline"; | |
| 22 | + | |
| 23 | +const runDir = resolve(process.argv[2] ?? ""); | |
| 24 | +const castPath = join(runDir, "terminal.cast"); | |
| 25 | +const browserPath = join(runDir, "session.mp4"); | |
| 26 | +if (!existsSync(castPath) || !existsSync(browserPath)) { | |
| 27 | + console.error(`film: need terminal.cast + session.mp4 in ${runDir}`); | |
| 28 | + process.exit(1); | |
| 29 | +} | |
| 30 | + | |
| 31 | +const run = (cmd: string, args: string[]) => execFileSync(cmd, args, { stdio: "pipe" }); | |
| 32 | +const probeSeconds = (file: string): number => | |
| 33 | + Number( | |
| 34 | + execFileSync("ffprobe", [ | |
| 35 | + ...["-v", "quiet", "-show_entries", "format=duration"], | |
| 36 | + ...["-of", "csv=p=0", file], | |
| 37 | + ]) | |
| 38 | + .toString() | |
| 39 | + .trim(), | |
| 40 | + ); | |
| 41 | + | |
| 42 | +// --------------------------------------------------------------------------- | |
| 43 | +// Build the cut list: acts of { source, from, to } in each source's clock. | |
| 44 | +// --------------------------------------------------------------------------- | |
| 45 | + | |
| 46 | +interface CastEvent { | |
| 47 | + readonly at: number; | |
| 48 | + readonly text: string; | |
| 49 | +} | |
| 50 | + | |
| 51 | +const events: CastEvent[] = readFileSync(castPath, "utf8") | |
| 52 | + .split("\n") | |
| 53 | + .slice(1) | |
| 54 | + .filter(Boolean) | |
| 55 | + .map((line) => JSON.parse(line) as [number, string, string]) | |
| 56 | + .filter((event) => event[1] === "o") | |
| 57 | + .map((event) => ({ at: event[0], text: event[2] })); | |
| 58 | + | |
| 59 | +const castEnd = events.at(-1)?.at ?? 0; | |
| 60 | + | |
| 61 | +interface Act { | |
| 62 | + readonly source: "terminal" | "browser"; | |
| 63 | + readonly from: number; | |
| 64 | + readonly to: number; | |
| 65 | +} | |
| 66 | + | |
| 67 | +/** Acts from the focus timeline: each contiguous focus run becomes a cut of | |
| 68 | + * that window's recording, mapped via the recording-start anchors. */ | |
| 69 | +const actsFromTimeline = (): Act[] | null => { | |
| 70 | + const timeline = readTimeline(runDir); | |
| 71 | + if (!timeline || timeline.focus.length < 2) return null; | |
| 72 | + const terminalAnchor = timeline.anchors.terminal; | |
| 73 | + const browserAnchor = timeline.anchors.browser; | |
| 74 | + if (terminalAnchor === undefined || browserAnchor === undefined) return null; | |
| 75 | + | |
| 76 | + const browserEnd = probeSeconds(browserPath); | |
| 77 | + const toMedia = (window: Act["source"], wallMs: number) => | |
| 78 | + Math.max(0, (wallMs - (window === "terminal" ? terminalAnchor : browserAnchor)) / 1000); | |
| 79 | + | |
| 80 | + const acts: Act[] = []; | |
| 81 | + for (let i = 0; i < timeline.focus.length; i += 1) { | |
| 82 | + const current = timeline.focus[i]; | |
| 83 | + if (!current) continue; | |
| 84 | + const nextAt = timeline.focus[i + 1]?.at; | |
| 85 | + const from = i === 0 && current.window === "terminal" ? 0 : toMedia(current.window, current.at); | |
| 86 | + const mediaEnd = current.window === "terminal" ? castEnd : browserEnd; | |
| 87 | + const to = | |
| 88 | + nextAt === undefined ? mediaEnd : Math.min(toMedia(current.window, nextAt), mediaEnd); | |
| 89 | + if (to - from > 0.5) acts.push({ source: current.window, from, to }); | |
| 90 | + } | |
| 91 | + return acts.length >= 2 ? acts : null; | |
| 92 | +}; | |
| 93 | + | |
| 94 | +/** Fallback for runs without a timeline: one browser hop located by the | |
| 95 | + * narrator line, or the largest output gap in the cast. */ | |
| 96 | +const actsFromHeuristic = (): Act[] => { | |
| 97 | + const findHop = (): { start: number; end: number } => { | |
| 98 | + const markerIndex = events.findIndex((event) => event.text.includes("in the browser")); | |
| 99 | + if (markerIndex !== -1) { | |
| 100 | + const start = events[markerIndex]; | |
| 101 | + const after = events | |
| 102 | + .slice(markerIndex + 1) | |
| 103 | + .find((event) => event.at > (start?.at ?? 0) + 1 && event.text.trim().length > 0); | |
| 104 | + if (start && after) return { start: start.at + 0.8, end: after.at }; | |
| 105 | + } | |
| 106 | + let best = { start: 0, end: 0 }; | |
| 107 | + for (let i = 1; i < events.length; i += 1) { | |
| 108 | + const previous = events[i - 1]; | |
| 109 | + const current = events[i]; | |
| 110 | + if (previous && current && current.at - previous.at > best.end - best.start) { | |
| 111 | + best = { start: previous.at, end: current.at }; | |
| 112 | + } | |
| 113 | + } | |
| 114 | + return best; | |
| 115 | + }; | |
| 116 | + const hop = findHop(); | |
| 117 | + if (hop.end - hop.start < 2) { | |
| 118 | + console.error(`film: no browser hop found in the cast (gap ${hop.end - hop.start}s)`); | |
| 119 | + process.exit(1); | |
| 120 | + } | |
| 121 | + return [ | |
| 122 | + { source: "terminal", from: 0, to: hop.start }, | |
| 123 | + { source: "browser", from: 0, to: probeSeconds(browserPath) }, | |
| 124 | + { source: "terminal", from: hop.end, to: castEnd }, | |
| 125 | + ]; | |
| 126 | +}; | |
| 127 | + | |
| 128 | +const acts = actsFromTimeline() ?? actsFromHeuristic(); | |
| 129 | + | |
| 130 | +// --------------------------------------------------------------------------- | |
| 131 | +// Render + cut + concatenate onto one canvas — full-screen cuts, like | |
| 132 | +// tabbing. | |
| 133 | +// --------------------------------------------------------------------------- | |
| 134 | + | |
| 135 | +const work = mkdtempSync(join(tmpdir(), "e2e-film-")); | |
| 136 | +const castGif = join(work, "cast.gif"); | |
| 137 | +const castVideo = join(work, "cast.mp4"); | |
| 138 | + | |
| 139 | +run("agg", [ | |
| 140 | + ...["--idle-time-limit", String(Math.ceil(castEnd) + 60)], | |
| 141 | + ...["--font-size", "16"], | |
| 142 | + castPath, | |
| 143 | + castGif, | |
| 144 | +]); | |
| 145 | +run("ffmpeg", [ | |
| 146 | + ...["-y", "-i", castGif], | |
| 147 | + // agg's gif can have odd dimensions; libx264 requires even. | |
| 148 | + ...["-vf", "scale=ceil(iw/2)*2:ceil(ih/2)*2"], | |
| 149 | + ...["-pix_fmt", "yuv420p", "-r", "24", castVideo], | |
| 150 | +]); | |
| 151 | + | |
| 152 | +const FIT = | |
| 153 | + "scale=1280:800:force_original_aspect_ratio=decrease,pad=1280:800:(ow-iw)/2:(oh-ih)/2:color=0x0b0b10,setsar=1,fps=24,format=yuv420p"; | |
| 154 | + | |
| 155 | +const filmPath = join(runDir, "film.mp4"); | |
| 156 | +const inputIndex = { terminal: 0, browser: 1 } as const; | |
| 157 | +const filters = acts.map( | |
| 158 | + (act, index) => | |
| 159 | + `[${inputIndex[act.source]}:v]trim=${act.from.toFixed(2)}:${act.to.toFixed(2)},setpts=PTS-STARTPTS,${FIT}[act${index}]`, | |
| 160 | +); | |
| 161 | +run("ffmpeg", [ | |
| 162 | + "-y", | |
| 163 | + ...["-i", castVideo], | |
| 164 | + ...["-i", browserPath], | |
| 165 | + "-filter_complex", | |
| 166 | + [ | |
| 167 | + ...filters, | |
| 168 | + `${acts.map((_, index) => `[act${index}]`).join("")}concat=n=${acts.length}:v=1:a=0[out]`, | |
| 169 | + ].join(";"), | |
| 170 | + ...["-map", "[out]"], | |
| 171 | + ...["-c:v", "libx264", "-preset", "veryfast", "-crf", "24", "-movflags", "+faststart"], | |
| 172 | + filmPath, | |
| 173 | +]); | |
| 174 | +rmSync(work, { recursive: true, force: true }); | |
| 175 | + | |
| 176 | +// Register the film in the run's artifact list so the viewer offers it. | |
| 177 | +const resultPath = join(runDir, "result.json"); | |
| 178 | +if (existsSync(resultPath)) { | |
| 179 | + const result = JSON.parse(readFileSync(resultPath, "utf8")) as { artifacts?: string[] }; | |
| 180 | + if (Array.isArray(result.artifacts) && !result.artifacts.includes("film.mp4")) { | |
| 181 | + result.artifacts.push("film.mp4"); | |
| 182 | + writeFileSync(resultPath, JSON.stringify(result, null, 1)); | |
| 183 | + } | |
| 184 | +} | |
| 185 | + | |
| 186 | +console.log( | |
| 187 | + `film: ${filmPath}\n${acts | |
| 188 | + .map( | |
| 189 | + (act, index) => | |
| 190 | + ` act ${index + 1} ${act.source} ${act.from.toFixed(1)}–${act.to.toFixed(1)}s`, | |
| 191 | + ) | |
| 192 | + .join("\n")}`, | |
| 193 | +); | |
| @@ -8,8 +8,9 @@ · 7 unmodified lines | ||
| 8 | 8 | // the PR description. |
| 9 | 9 | // |
| 10 | 10 | // Usage: bun e2e/scripts/pr-media.ts <run-dir-or-artifact> [...more] |
| 11 | −// run dir e2e/runs/<target>/<scenario-slug> — picks session.mp4 or | |
| 12 | −// terminal.cast and labels the gif from result.json | |
| 11 | +// run dir e2e/runs/<target>/<scenario-slug> — picks film.mp4 (the | |
| 12 | +// whole session), else session.mp4, else terminal.cast; | |
| 13 | +// labels the gif from result.json | |
| 13 | 14 | // session.mp4 browser recording (ffmpeg -> gif) |
| 14 | 15 | // terminal.cast terminal recording (agg -> gif; brew install agg) |
| 15 | 16 | // *.png run screenshot, uploaded as-is |
| @@ -48,10 +49,13 @@ const resolveArtifact = (input: string): Artifact => { · 32 unmodified lines | ||
| 48 | 49 | slug: basename(path).replace(/\.[^.]+$/, ""), |
| 49 | 50 | }; |
| 50 | 51 | } |
| 51 | − const recording = ["session.mp4", "terminal.cast"] | |
| 52 | + // film.mp4 (scripts/film.ts) is the whole session — terminal chat AND the | |
| 53 | + // browser hop, cut in time order; the bare browser session.mp4 is only the | |
| 54 | + // hop, so it must never win when a film exists. | |
| 55 | + const recording = ["film.mp4", "session.mp4", "terminal.cast"] | |
| 52 | 56 | .map((name) => join(path, name)) |
| 53 | 57 | .find(existsSync); |
| 54 | − if (!recording) throw new Error(`${input} has no session.mp4 or terminal.cast`); | |
| 58 | + if (!recording) throw new Error(`${input} has no film.mp4, session.mp4, or terminal.cast`); | |
| 55 | 59 | let label = basename(path); |
| 56 | 60 | try { |
| 57 | 61 | const result = JSON.parse(readFileSync(join(path, "result.json"), "utf8")) as { |