History

Make the E2E (stdio MCP) job robust to two harness flakes (#1172)

The "E2E (stdio MCP)" CI job runs `stdio-mcp.test.ts` alone, so it always pays a cold `vite optimizeDeps` boot, the one variable step. Two unrelated harness issues let that job go red without a real product regression. 1. Blind boot timeout. The scenario's test timeout (180s) was set equal to the boot-URL wait inside `withLocalServer` (180s). When boot is slow both deadlines fire together and vitest's generic "Test timed out in 180000ms" wins, so the failure carries no clue about what boot got stuck on. Raise the scenario timeout to 240s so the boot wait trips first, and have that wait reject with the terminal tail (waitUntil otherwise throws a tail-less "timed out"). A slow or wedged boot now reports how far it got. 2. Teardown crash on a truncated recording. `toAsciicast` JSON.parsed every recording line; a PTY killed mid-write leaves a truncated final JSONL line, throwing "Unexpected end of JSON input" and failing an otherwise-passing test over a best-effort film artifact. Skip unparseable lines (a dropped frame, not a failure) and make the cast write best-effort, matching the recording fetch beside it.

Rhys Sullivan committed 4 months ago

Commit d5ba68f1 parent 5250f643 files · +60 −18Browse files

Unified · Split

3 files
  1. e2e/local/local-server.ts +23 −6
  2. e2e/local/stdio-mcp.test.ts +8 −1
  3. e2e/src/surfaces/cli.ts +29 −11
e2e/local/local-server.ts+23−6
@@ -59,12 +59,29 @@ export const withLocalServer = ( · 58 unmodified lines
59 async (term) => {59 async (term) => {
60 markRecordingStart(runDir, "terminal");60 markRecordingStart(runDir, "terminal");
61 markFocus(runDir, "terminal");61 markFocus(runDir, "terminal");
62 const snapshot = await term.screen.waitUntil(62 // Capture the latest rendered screen on every poll so a boot that
63 (current) => TOKEN_URL.test(current.text),63 // never prints the URL is still diagnosable: `waitUntil` rejects
64 // A cold `executor web` runs vite's optimizeDeps before it prints64 // with a tail-less "timed out" on its deadline, so without this the
65 // the URL; give CI runners headroom over the warm ~3s boot.65 // failure tells us nothing about how far boot actually got.
66 { timeoutMs: 180_000 },66 let lastText = "";
67 );67 const snapshot = await term.screen
68 .waitUntil(
69 (current) => {
70 lastText = current.text;
71 return TOKEN_URL.test(current.text);
72 },
73 // A cold `executor web` runs vite's optimizeDeps before it prints
74 // the URL; 180s is generous headroom over the warm ~3s boot. Kept
75 // BELOW each scenario's test timeout (240s) so that on a slow or
76 // wedged boot THIS error surfaces, with the terminal tail, rather
77 // than racing vitest's generic timeout at the same deadline.
78 { timeoutMs: 180_000 },
79 )
80 .catch((cause: unknown) => {
81 throw new Error(
82 `executor web --foreground printed no ?_token URL within 180s (${String(cause)}):\n${lastText.slice(-600)}`,
83 );
84 });
68 const url = TOKEN_URL.exec(snapshot.text)?.[0];85 const url = TOKEN_URL.exec(snapshot.text)?.[0];
69 if (!url) {86 if (!url) {
70 throw new Error(87 throw new Error(
e2e/local/stdio-mcp.test.ts+8−1
@@ -39,7 +39,14 @@ const SECRET = "s3cr3t-from-the-vault"; · 38 unmodified lines
39 39
40scenario(40scenario(
41 "Local · a stdio MCP server's tools are detected on a fresh install, with env stored as a secret",41 "Local · a stdio MCP server's tools are detected on a fresh install, with env stored as a secret",
42 { timeout: 180_000 },42 // Must stay STRICTLY greater than the boot-URL wait in `withLocalServer`
43 // (currently 180s). When this CI job runs `stdio-mcp.test.ts` alone it always
44 // pays a cold `vite optimizeDeps` boot (no prior file to warm the cache), the
45 // one variable step. If this test timeout equals the boot wait, both deadlines
46 // fire together and vitest's generic "Test timed out" wins, swallowing the
47 // harness's "printed no ?_token URL\n<terminal tail>" error that tells us what
48 // boot actually got stuck on. Keep the gap so the boot diagnostic surfaces.
49 { timeout: 240_000 },
43 Effect.gen(function* () {50 Effect.gen(function* () {
44 const cli = yield* Cli;51 const cli = yield* Cli;
45 const runDir = yield* RunDir;52 const runDir = yield* RunDir;
e2e/src/surfaces/cli.ts+29−11
@@ -24,22 +24,32 @@ export interface CliSurface { · 23 unmodified lines
24 ) => Effect.Effect<T>;24 ) => Effect.Effect<T>;
25}25}
26 26
27interface AsciicastEvent {
28 type: string;
29 cols?: number;
30 rows?: number;
31 at_ms?: number;
32 bytes?: number[];
33}
34
27/** terminal-control's JSONL recording → asciicast v2 (what asciinema plays). */35/** terminal-control's JSONL recording → asciicast v2 (what asciinema plays). */
28const toAsciicast = (recording: Uint8Array): string => {36const toAsciicast = (recording: Uint8Array): string => {
29 const events = new TextDecoder()37 const events = new TextDecoder()
30 .decode(recording)38 .decode(recording)
31 .split("\n")39 .split("\n")
32 .filter(Boolean)40 .filter(Boolean)
33 .map(41 .flatMap((line): AsciicastEvent[] => {
34 (line) =>42 // The PTY can be killed mid-write on teardown (e.g. a vitest timeout
35 JSON.parse(line) as {43 // interrupts the fiber), leaving a truncated final JSONL line. A line
36 type: string;44 // that doesn't parse is one dropped frame, never a failed test: skip it
37 cols?: number;45 // instead of throwing.
38 rows?: number;46 // oxlint-disable-next-line executor/no-try-catch-or-throw -- boundary: lenient parse of a best-effort recording artifact
39 at_ms?: number;47 try {
40 bytes?: number[];48 return [JSON.parse(line) as AsciicastEvent];
41 },49 } catch {
42 );50 return [];
51 }
52 });
43 const header = events.find((event) => event.type === "header");53 const header = events.find((event) => event.type === "header");
44 const lines = [54 const lines = [
45 JSON.stringify({ version: 2, width: header?.cols ?? 80, height: header?.rows ?? 24 }),55 JSON.stringify({ version: 2, width: header?.cols ?? 80, height: header?.rows ?? 24 }),
@@ -89,7 +99,15 @@ export const makeCliSurface = (): CliSurface => ({ · 43 unmodified lines
89 Effect.promise(async () => {99 Effect.promise(async () => {
90 if (options?.record) {100 if (options?.record) {
91 const recording = await session.recording().catch(() => undefined);101 const recording = await session.recording().catch(() => undefined);
92 if (recording) writeFileSync(options.record, toAsciicast(recording));102 // Writing the cast is a best-effort film artifact (symmetric with the
103 // fetch above): a serialization or IO hiccup in teardown must never
104 // fail an otherwise-passing test.
105 // oxlint-disable-next-line executor/no-try-catch-or-throw -- boundary: best-effort artifact write
106 try {
107 if (recording) writeFileSync(options.record, toAsciicast(recording));
108 } catch {
109 // drop the cast; the test result stands
110 }
93 }111 }
94 await session.stop().catch(() => {});112 await session.stop().catch(() => {});
95 await tc[Symbol.asyncDispose]();113 await tc[Symbol.asyncDispose]();

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t