Make e2e port claiming atomic: lock port per block, walk past collisions

The hash-derived block was only collision-unlikely (28 checkouts over 400 blocks is birthday-paradox territory), and probe-then-bind raced. Each block now reserves its last port as a lock held for the suite's lifetime: claimPorts binds the lock (atomic — two racing suites can't both win), probes the remaining ports for squatters, and walks forward block-by-block until it owns a fully free block, publishing the claimed ports via E2E_*_PORT env so test workers agree. Explicit env pins skip claiming entirely. Verified by squatting the preferred block's lock+selfhost ports and watching the suite relocate one block over and pass 17/17.

Rhys Sullivan committed 5 months ago

Commit 3f3ed8f1 parent acb96656 files · +183 −86Browse files

Unified · Split

6 files

e2e/AGENTS.md +7 −4
e2e/scripts/ports.ts +6 −3
e2e/setup/boot.ts +1 −39
e2e/setup/cloud.globalsetup.ts +26 −24
e2e/setup/selfhost.globalsetup.ts +20 −7
e2e/src/ports.ts +123 −9

e2e/AGENTS.md+7−4

 E2E_SELFHOST_URL=http://localhost:<port> ../node_modules/.bin/vitest run --project selfhost <file>
 ```
-Ports are derived per checkout (hash of the repo root — see `src/ports.ts`),
+Ports are claimed at boot (see `src/ports.ts`): each checkout hashes its repo
-so suites in different worktrees never fight. If a port is somehow taken, the
+root to a preferred block, atomically locks it (a held lock port makes races
-boot fails fast naming the squatting process; kill it or override with
+impossible), and walks to the next free block if it's locked or squatted — so
-`E2E_CLOUD_PORT`-style env vars.
+concurrent suites in different worktrees can never collide or attach to each
+other's servers. `bun run ports` shows the preferred block; the boot log says
+if a suite moved. `E2E_*_PORT` env vars pin ports explicitly (no probing) and
+`E2E_<TARGET>_URL` attaches to a running instance.
 Each run writes `runs/<target>/<slug>/result.json` plus any browser artifacts
 (trace.zip / session.mp4 / screenshots). `bun run serve` hosts the scenario ×

e2e/scripts/ports.ts+6−3

-// Print this checkout's derived e2e ports (see src/ports.ts) so an agent or
+// Print this checkout's PREFERRED e2e ports (see src/ports.ts). These are
-// human can curl the booted servers or attach with E2E_<TARGET>_URL.
+// where a suite normally boots; if the block is locked or squatted at boot
+// time, claimPorts walks to the next free block and the suite logs the move.
+// When attaching mid-run, the booted vite's actual port is authoritative —
+// check the suite's log line or `ps | grep 'vite dev'`.
 import {
   AUTUMN_EMULATOR_PORT,
   CLOUD_DB_PORT,
 import { SELFHOST_PORT } from "../targets/selfhost";
 import { repoRoot } from "../src/ports";
-console.log(`e2e ports for ${repoRoot}`);
+console.log(`preferred e2e ports for ${repoRoot}`);
 console.log(`  cloud           http://127.0.0.1:${CLOUD_PORT}`);
 console.log(`  cloud dev-db    ${CLOUD_DB_PORT}`);
 console.log(`  workos emulator ${WORKOS_EMULATOR_PORT}`);

e2e/setup/boot.ts+1−39

@@ -2,8 +2,7 @@ · 1 unmodified line
2	// server, wait until it answers HTTP, and hand vitest a teardown. The apps own	2	// server, wait until it answers HTTP, and hand vitest a teardown. The apps own
3	// what runs (their dev stack, their stub flags); this file only owns process	3	// what runs (their dev stack, their stub flags); this file only owns process
4	// lifecycle, so it stays target-agnostic.	4	// lifecycle, so it stays target-agnostic.
5	import { execFileSync, spawn, type ChildProcess } from "node:child_process";	5	import { spawn, type ChildProcess } from "node:child_process";
6	import { connect } from "node:net";
7		6
8	export interface BootedProcesses {	7	export interface BootedProcesses {
9	readonly teardown: () => Promise<void>;	8	readonly teardown: () => Promise<void>;
@@ -73,43 +72,6 @@ export const bootProcesses = ( · 63 unmodified lines
73	};	72	};
74	};	73	};
75		74
76	// A port already in LISTEN before we boot means waitForHttp would silently
77	// attach to a FOREIGN server (a leaked dev server from another checkout or a
78	// crashed prior run) — every scenario then fails with baffling auth errors
79	// instead of one clear message. Fail fast and name the squatter.
80	export const ensurePortsFree = async (
81	ports: ReadonlyArray<{ readonly port: number; readonly label: string }>,
82	): Promise<void> => {
83	for (const { port, label } of ports) {
84	const inUse = await new Promise<boolean>((resolve) => {
85	const socket = connect({ port, host: "127.0.0.1" });
86	socket.once("connect", () => {
87	socket.destroy();
88	resolve(true);
89	});
90	socket.once("error", () => resolve(false));
91	socket.setTimeout(1_000, () => {
92	socket.destroy();
93	resolve(false);
94	});
95	});
96	if (!inUse) continue;
97	let owner = "(lsof unavailable)";
98	try {
99	owner = execFileSync("lsof", ["-nP", `-iTCP:${port}`, "-sTCP:LISTEN"], {
100	encoding: "utf8",
101	}).trim();
102	} catch {
103	// lsof failing is fine — the error below carries the essential fact.
104	}
105	throw new Error(
106	`e2e: port ${port} (${label}) is already in use — likely a leaked dev server ` +
107	`from another checkout or a previous run. Kill it or set the E2E_*_PORT/URL ` +
108	`env vars to relocate.\n${owner}`,
109	);
110	}
111	};
112
113	export const waitForHttp = async (	75	export const waitForHttp = async (
114	url: string,	76	url: string,
115	options: { readonly timeoutMs?: number; readonly expectRedirect?: boolean } = {},	77	options: { readonly timeoutMs?: number; readonly expectRedirect?: boolean } = {},

e2e/setup/cloud.globalsetup.ts+26−24

 // Vendored fork import (same pattern as mcporter).
 import { createEmulator } from "@executor-js/emulate";
-import { bootProcesses, ensurePortsFree, waitForHttp } from "./boot";
+import { claimPorts } from "../src/ports";
-import {
+import { E2E_COOKIE_PASSWORD, E2E_WORKOS_CLIENT_ID } from "../targets/cloud";
-  CLOUD_BASE_URL,
+import { bootProcesses, waitForHttp } from "./boot";
-  CLOUD_DB_PORT,
-  CLOUD_PORT,
-  WORKOS_EMULATOR_PORT,
-  AUTUMN_EMULATOR_PORT,
-  E2E_WORKOS_CLIENT_ID,
-  E2E_COOKIE_PASSWORD,
-} from "../targets/cloud";
 const cloudDir = fileURLToPath(new URL("../../apps/cloud/", import.meta.url));
     return;
   }
-  await ensurePortsFree([
+  // Claim a free port block (preferred block first, walk forward past
-    { port: CLOUD_PORT, label: "cloud vite dev" },
+  // squatters/colliding checkouts) and publish via env so the test workers —
-    { port: CLOUD_DB_PORT, label: "cloud dev-db (PGlite)" },
+  // spawned after this — derive the same URLs. The imported targets/cloud
-    { port: WORKOS_EMULATOR_PORT, label: "WorkOS emulator" },
+  // constants were computed BEFORE the claim, so use the claimed values here.
-    { port: AUTUMN_EMULATOR_PORT, label: "Autumn emulator" },
+  const { ports, release } = await claimPorts([
+    { envVar: "E2E_CLOUD_PORT", offset: 0, label: "cloud vite dev" },
+    { envVar: "E2E_CLOUD_DB_PORT", offset: 1, label: "cloud dev-db (PGlite)" },
+    { envVar: "E2E_WORKOS_EMULATOR_PORT", offset: 2, label: "WorkOS emulator" },
+    { envVar: "E2E_AUTUMN_EMULATOR_PORT", offset: 3, label: "Autumn emulator" },
   ]);
+  const cloudPort = ports.E2E_CLOUD_PORT!;
+  const dbPort = ports.E2E_CLOUD_DB_PORT!;
+  const baseUrl = `http://127.0.0.1:${cloudPort}`;
   // Fresh dev DB per suite run — hermetic, like the selfhost data dir. The
   // WorkOS emulator mints org ids from a per-process counter, so a persisted
   // MCP access tokens minted by the emulator's OAuth server must carry the
   // app's client id as audience (what the resource server verifies).
   process.env.EMULATE_WORKOS_AUDIENCE = E2E_WORKOS_CLIENT_ID;
-  const workos = await createEmulator({ service: "workos", port: WORKOS_EMULATOR_PORT });
+  const workos = await createEmulator({ service: "workos", port: ports.E2E_WORKOS_EMULATOR_PORT! });
-  const autumn = await createEmulator({ service: "autumn", port: AUTUMN_EMULATOR_PORT });
+  const autumn = await createEmulator({ service: "autumn", port: ports.E2E_AUTUMN_EMULATOR_PORT! });
   const env = {
     // Real client, emulated service.
     WORKOS_COOKIE_PASSWORD: E2E_COOKIE_PASSWORD,
     AUTUMN_SECRET_KEY: "am_test_emulate",
     ENCRYPTION_KEY: "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
-    DATABASE_URL: `postgresql://postgres:postgres@127.0.0.1:${CLOUD_DB_PORT}/postgres`,
+    DATABASE_URL: `postgresql://postgres:postgres@127.0.0.1:${dbPort}/postgres`,
     EXECUTOR_DIRECT_DATABASE_URL: "true",
     CLOUDFLARE_INCLUDE_PROCESS_ENV: "true",
-    VITE_PUBLIC_SITE_URL: CLOUD_BASE_URL,
+    VITE_PUBLIC_SITE_URL: baseUrl,
     // The AuthKit domain (MCP OAuth metadata + JWKS) is the emulator too.
     MCP_AUTHKIT_DOMAIN: workos.url,
-    MCP_RESOURCE_ORIGIN: CLOUD_BASE_URL,
+    MCP_RESOURCE_ORIGIN: baseUrl,
     ALLOW_LOCAL_NETWORK: "true",
     // Throwaway PGlite on its own port + dir so it never fights `bun dev`.
-    DEV_DB_PORT: String(CLOUD_DB_PORT),
+    DEV_DB_PORT: String(dbPort),
     DEV_DB_PATH: dbPath,
   };
       { cmd: "bun", args: ["run", "scripts/dev-db.ts"], cwd: cloudDir, env },
       {
         cmd: "bunx",
-        args: ["vite", "dev", "--port", String(CLOUD_PORT), "--strictPort", "--host", "127.0.0.1"],
+        args: ["vite", "dev", "--port", String(cloudPort), "--strictPort", "--host", "127.0.0.1"],
         cwd: cloudDir,
         env,
       },
   );
   try {
-    await waitForHttp(CLOUD_BASE_URL);
+    await waitForHttp(baseUrl);
     // The API plane is ready when login actually redirects to AuthKit.
-    await waitForHttp(`${CLOUD_BASE_URL}/api/auth/login`, { expectRedirect: true });
+    await waitForHttp(`${baseUrl}/api/auth/login`, { expectRedirect: true });
   } catch (error) {
     await procs.teardown();
     await workos.close();
     await autumn.close();
+    await release();
     throw error;
   }
   return async () => {
     await procs.teardown();
     await workos.close();
     await autumn.close();
+    await release();
   };
 }

e2e/setup/selfhost.globalsetup.ts+20−7

 import { resolve } from "node:path";
 import { fileURLToPath } from "node:url";
-import { bootProcesses, ensurePortsFree, waitForHttp } from "./boot";
+import { claimPorts } from "../src/ports";
-import { SELFHOST_ADMIN, SELFHOST_BASE_URL, SELFHOST_PORT } from "../targets/selfhost";
+import { SELFHOST_ADMIN } from "../targets/selfhost";
+import { bootProcesses, waitForHttp } from "./boot";
 const selfhostDir = fileURLToPath(new URL("../../apps/host-selfhost/", import.meta.url));
     return;
   }
-  await ensurePortsFree([{ port: SELFHOST_PORT, label: "selfhost vite dev" }]);
+  // Claim a free port (preferred block first, walk forward past squatters)
+  // and publish via env so the test workers derive the same URL. The imported
+  // targets/selfhost constants were computed BEFORE the claim — don't use them
+  // for ports/URLs here.
+  const { ports, release } = await claimPorts([
+    { envVar: "E2E_SELFHOST_PORT", offset: 4, label: "selfhost vite dev" },
+  ]);
+  const port = ports.E2E_SELFHOST_PORT!;
+  const baseUrl = `http://localhost:${port}`;
   // Fresh data dir per suite run — hermetic; in-suite isolation comes from
   // fresh identities, not resets.
     [
       {
         cmd: "bunx",
-        args: ["--bun", "vite", "dev", "--port", String(SELFHOST_PORT), "--strictPort"],
+        args: ["--bun", "vite", "dev", "--port", String(port), "--strictPort"],
         cwd: selfhostDir,
         env: {
           EXECUTOR_DATA_DIR: dataDir,
           BETTER_AUTH_SECRET: "executor-selfhost-e2e-secret-0123456789",
           EXECUTOR_BOOTSTRAP_ADMIN_EMAIL: SELFHOST_ADMIN.email,
           EXECUTOR_BOOTSTRAP_ADMIN_PASSWORD: SELFHOST_ADMIN.password,
-          EXECUTOR_WEB_BASE_URL: SELFHOST_BASE_URL,
+          EXECUTOR_WEB_BASE_URL: baseUrl,
           // The harness boots loopback MCP/OAuth test servers and points the
           // instance at them; the hosted SSRF guard would otherwise block
           // outbound probes/dials to localhost. Hermetic test instance only.
   );
   try {
-    await waitForHttp(SELFHOST_BASE_URL);
+    await waitForHttp(baseUrl);
   } catch (error) {
     await procs.teardown();
+    await release();
     throw error;
   }
-  return procs.teardown;
+  return async () => {
+    await procs.teardown();
+    await release();
+  };
 }

e2e/src/ports.ts+123−9

@@ -1,11 +1,18 @@
1	// Per-checkout port derivation: every checkout (main repo, agent worktree,	1	// Per-checkout port derivation: every checkout (main repo, agent worktree,
2	// /tmp rig) gets its own deterministic block of e2e ports, so concurrent	2	// /tmp rig) hashes its repo root into a PREFERRED block of e2e ports, so
3	// suites never fight over a shared default. The collision failure mode is	3	// concurrent suites normally never fight over a shared default. The hash is
4	// brutal: vite's --strictPort exit is swallowed by the boot glue and	4	// only a preference, not a guarantee (28 checkouts over 400 blocks is
5	// waitForHttp happily attaches to the OTHER checkout's server, failing dozens	5	// birthday-paradox territory) — the globalsetups call `claimPorts`, which
6	// of scenarios with baffling auth errors instead of one clear bind error.	6	// probes the preferred block and walks forward to the next fully-free one,
7	// Individual E2E_*_PORT env vars still override, and E2E_<TARGET>_URL still	7	// then publishes the claimed ports via the E2E_*_PORT env vars so vitest's
8	// attaches to a running instance.	8	// test workers (spawned after globalsetup) compute the same URLs. The
		9	// collision failure mode this kills is brutal: vite's --strictPort exit is
		10	// swallowed by the boot glue and waitForHttp happily attaches to the OTHER
		11	// checkout's server, failing dozens of scenarios with baffling auth errors
		12	// instead of one clear bind error. Individual E2E_*_PORT env vars still
		13	// override everything, and E2E_<TARGET>_URL still attaches to a running
		14	// instance.
		15	import { connect, createServer, type Server } from "node:net";
9	import { resolve } from "node:path";	16	import { resolve } from "node:path";
10	import { fileURLToPath } from "node:url";	17	import { fileURLToPath } from "node:url";
11		18
@@ -24,10 +31,117 @@ const hash = (text: string): number => { · 12 unmodified lines
24	};	31	};
25		32
26	// 400 blocks of 10 ports in 42000..45999: unprivileged, clear of common dev	33	// 400 blocks of 10 ports in 42000..45999: unprivileged, clear of common dev
27	// servers, and below macOS's ephemeral range (49152+).	34	// servers, and below macOS's ephemeral range (49152+). Offsets 0-8 are
28	export const portBlock = 42000 + (hash(repoRoot) % 400) * 10;	35	// claimable; offset 9 is the block's lock port (held for the suite's
		36	// lifetime to make claims atomic across concurrent suites).
		37	const BLOCK_BASE = 42000;
		38	const BLOCK_SIZE = 10;
		39	const BLOCK_COUNT = 400;
		40	const LOCK_OFFSET = BLOCK_SIZE - 1;
		41	export const portBlock = BLOCK_BASE + (hash(repoRoot) % BLOCK_COUNT) * BLOCK_SIZE;
29		42
30	export const e2ePort = (envVar: string, offset: number): number => {	43	export const e2ePort = (envVar: string, offset: number): number => {
31	const fromEnv = process.env[envVar];	44	const fromEnv = process.env[envVar];
32	return fromEnv ? Number(fromEnv) : portBlock + offset;	45	return fromEnv ? Number(fromEnv) : portBlock + offset;
33	};	46	};
		47
		48	const isListening = (port: number): Promise<boolean> =>
		49	new Promise((done) => {
		50	const socket = connect({ port, host: "127.0.0.1" });
		51	socket.once("connect", () => {
		52	socket.destroy();
		53	done(true);
		54	});
		55	socket.once("error", () => done(false));
		56	socket.setTimeout(1_000, () => {
		57	socket.destroy();
		58	done(false);
		59	});
		60	});
		61
		62	export interface PortClaim {
		63	readonly envVar: string;
		64	readonly offset: number;
		65	readonly label: string;
		66	}
		67
		68	export interface ClaimedPorts {
		69	readonly ports: Record<string, number>;
		70	/** Releases the block's lock port — call from the suite teardown. */
		71	readonly release: () => Promise<void>;
		72	}
		73
		74	// Binding is atomic where probing is not: holding the block's lock port for
		75	// the suite's lifetime means two suites racing for the same block can never
		76	// both win (the second bind EADDRINUSEs and walks on).
		77	const tryLockBlock = (block: number): Promise<Server \| undefined> =>
		78	new Promise((done) => {
		79	const server = createServer();
		80	server.once("error", () => done(undefined));
		81	server.listen(block + LOCK_OFFSET, "127.0.0.1", () => done(server));
		82	});
		83
		84	/**
		85	* Claim a free set of ports for a target and publish them via env. Starts at
		86	* this checkout's preferred block and walks forward block-by-block until it
		87	* can atomically lock a block whose requested ports are all free — so two
		88	* checkouts whose hashes collide (or a leaked server squatting the preferred
		89	* block) degrade to "boot one block over" instead of attaching to a foreign
		90	* server. Explicit env overrides win and are never probed or locked: if you
		91	* pin a port and it's busy, vite's --strictPort fails visibly. A target
		92	* re-claiming inside an already-locked process (cloud + selfhost projects in
		93	* one vitest run) shares the block via disjoint offsets.
		94	*/
		95	export const claimPorts = async (claims: ReadonlyArray<PortClaim>): Promise<ClaimedPorts> => {
		96	const ports: Record<string, number> = {};
		97	const unpinned = claims.filter((claim) => {
		98	const pinned = process.env[claim.envVar];
		99	if (pinned) ports[claim.envVar] = Number(pinned);
		100	return !pinned;
		101	});
		102	if (unpinned.length === 0) return { ports, release: async () => {} };
		103
		104	for (let attempt = 0; attempt < BLOCK_COUNT; attempt++) {
		105	const block =
		106	BLOCK_BASE + ((portBlock - BLOCK_BASE + attempt * BLOCK_SIZE) % (BLOCK_COUNT * BLOCK_SIZE));
		107	// This process may already hold the block's lock (the other target's
		108	// globalsetup in the same vitest run); reuse it instead of re-locking.
		109	let lock = heldLocks.get(block);
		110	if (!lock) {
		111	lock = await tryLockBlock(block);
		112	if (!lock) {
		113	console.warn(`[e2e] port block ${block} is locked by another suite; trying next block`);
		114	continue;
		115	}
		116	heldLocks.set(block, lock);
		117	}
		118	const busy = await Promise.all(unpinned.map((claim) => isListening(block + claim.offset)));
		119	if (busy.some(Boolean)) {
		120	const taken = unpinned
		121	.filter((_, index) => busy[index])
		122	.map((claim) => `${block + claim.offset} (${claim.label})`);
		123	console.warn(
		124	`[e2e] port block ${block} has squatters — ${taken.join(", ")}; trying next block`,
		125	);
		126	continue; // Keep the lock: a half-busy block is still ours, just unusable now.
		127	}
		128	for (const claim of unpinned) {
		129	const port = block + claim.offset;
		130	ports[claim.envVar] = port;
		131	// Workers spawn after globalsetup, so they inherit these and agree.
		132	process.env[claim.envVar] = String(port);
		133	}
		134	return {
		135	ports,
		136	release: async () => {
		137	const held = heldLocks.get(block);
		138	if (!held) return;
		139	heldLocks.delete(block);
		140	await new Promise<void>((done) => held.close(() => done()));
		141	},
		142	};
		143	}
		144	throw new Error("e2e: no free port block found — the 42000-45999 range is exhausted?");
		145	};
		146
		147	const heldLocks = new Map<number, Server>();

Make e2e port claiming atomic: lock port per block, walk past collisions

Keyboard shortcuts

General

Issues and pull requests

Issues, pull requests and Inbox

Repository code