History

e2e: cross-OS supervised-daemon reboot survival (real VMs, recorded) (#1013)

* e2e: VM substrate for cross-OS supervised-daemon targets + build --target Foundation for codifying the supervised-daemon reboot-survival proof as e2e tests (today it's only a by-hand bash harness). Adds a VM substrate that can provision a guest, drive it over SSH, REBOOT the OS for real, and tear it down: - e2e/src/vm/{types,tart,build-binary}.ts — the tart provider (macOS + Linux guests on an Apple-Silicon host) with a reconnecting SSH forward that survives a guest reboot, plus a helper to compile the guest's `executor` binary. - apps/cli/src/build.ts: a first-class `--target <name>` flag so the harness can build a specific platform's binary (replaces a throwaway env hack). Validated standalone: provision macOS guest → ssh → real reboot (boottime changes) → clean discard. Next: the `cli` target wires this into the scenario suite so restart-persistence runs against a real reboot. * e2e: cli target — supervised daemon in a VM, restart() is a real reboot Wires the VM substrate into the scenario suite as a `cli` target (one project per guest OS: cli-macos / cli-linux / cli-windows). The globalsetup builds the guest's executor binary, provisions a VM, `executor service install`s the supervised daemon, and forwards its loopback port over a reconnecting SSH tunnel — so target.baseUrl works unchanged for the api surface. restart() reboots the guest OS for REAL and waits for the daemon to auto-start, so the existing restart-persistence scenario now proves the boot-time auto-start path (launchd RunAtLoad / systemd linger / Task Scheduler AtStartup), not a process restart. Because globalsetup (main process) and restart() (test worker) are different processes, restart() re-derives the guest address from env and reboots statelessly, mirroring how selfhost-docker's restart re-derives container ops. Verified: `vitest run --project cli-macos` boots a macOS guest, installs the daemon, addSpec's an integration, reboots the guest, and asserts the data survived — green, with a run-record in the matrix. cli-linux reuses the tart provider; cli-windows awaits the ec2 provider. * e2e: recorded film — integration survives a real reboot (terminal cast) A watchable companion to restart-persistence: drives the same register → reboot → survived flow through the chat theater so the on-screen tool spinner runs for the ACTUAL guest reboot. Produces a terminal.cast the viewer plays in the matrix — press-play evidence instead of trusting a green check. Runs against the cli-* VM targets; verified on cli-macos (357-event cast). * e2e: desktop attach film — app connects to the supervised daemon (not a sidecar) Films the desktop ATTACHING to an already-running OS-supervised daemon instead of spawning its own sidecar: starts a real cli-daemon (the desktop sidecar server in EXECUTOR_SUPERVISED mode → kind "cli-daemon") against a throwaway HOME, launches Electron pointed at the same HOME, and proves the attach — the server manifest still names OUR daemon's pid (a spawned sidecar would be a new pid + kind "desktop-sidecar"). session.mp4 is the artifact. The daemon half is verified (EXECUTOR_READY + kind cli-daemon + 401). Like the existing desktop scenario it drives Electron via Playwright, so it needs a GUI display session to run (the desktop project is already display-gated, not in the default test chain) — run on a real desktop / CI-with-display to capture the video. * e2e/cli: bearer-gated daemon target, reboot down-gate, EC2 provider for Windows - cli target authenticates with the daemon's auth.json bearer (was Basic from service.key); globalsetup reads the token from the guest and publishes it. - restart() gates on the daemon going DOWN before polling it back up, so an orderly shutdown / reconnecting tunnel can't false-pass a reboot that never happened (waitForHttpDown). - EC2 VmProvider (e2e/src/vm/ec2.ts) + cli-windows globalsetup unblock Windows: launch, key-based SSH/PowerShell drive, real boot-time-gated reboot, terminate. * e2e/desktop + cli tests: bearer auth token, not a password The supervised-attach film passed EXECUTOR_AUTH_PASSWORD, a dead env the sidecar no longer reads (it reads EXECUTOR_AUTH_TOKEN and otherwise mints/loads the bearer from auth.json). Pin a real bearer instead. Fix two stale launchd/wrapper comments that still pointed the secret at service.key.

Rhys Sullivan committed 5 months ago

Commit 211fa521 parent 956417416 files · +1273 −6Browse files

Unified · Split

16 files
  1. apps/cli/src/build.ts +19 −4
  2. apps/cli/src/service.test.ts +2 −2
  3. e2e/cli/service-lifecycle.test.ts +128 −0
  4. e2e/desktop/supervised-attach.test.ts +182 −0
  5. e2e/setup/boot.ts +24 −0
  6. e2e/setup/cli-linux.globalsetup.ts +3 −0
  7. e2e/setup/cli-macos.globalsetup.ts +3 −0
  8. e2e/setup/cli-windows.globalsetup.ts +3 −0
  9. e2e/setup/cli.globalsetup.ts +97 −0
  10. e2e/src/vm/build-binary.ts +35 −0
  11. e2e/src/vm/ec2.ts +422 −0
  12. e2e/src/vm/tart.ts +224 −0
  13. e2e/src/vm/types.ts +51 −0
  14. e2e/targets/cli.ts +59 −0
  15. e2e/targets/registry.ts +6 −0
  16. e2e/vitest.config.ts +15 −0
apps/cli/src/build.ts+19−4
@@ -501,18 +501,24 @@ const buildPreviewTarballs = async (binaries: Record<string, string>) => { · 500 unmodified lines
501 }501 }
502};502};
503 503
504// Resolve a comma-separated list of target package names (e.g.
505// "executor-windows-x64") to Targets. Shared by `--target` and the
506// EXECUTOR_PREVIEW_TARGETS env used by the preview-wrapper CI job.
504const resolveTargetsFromEnv = (env: string | undefined): Target[] => {507const resolveTargetsFromEnv = (env: string | undefined): Target[] => {
505 if (!env) throw new Error("EXECUTOR_PREVIEW_TARGETS must be set (comma-separated package names)");508 if (!env) throw new Error("No build targets given (comma-separated package names)");
506 const names = env509 const names = env
507 .split(",")510 .split(",")
508 .map((s) => s.trim())511 .map((s) => s.trim())
509 .filter(Boolean);512 .filter(Boolean);
510 const resolved = names.map((name) => {513 const resolved = names.map((name) => {
511 const match = ALL_TARGETS.find((t) => targetPackageName(t) === name);514 const match = ALL_TARGETS.find((t) => targetPackageName(t) === name);
512 if (!match) throw new Error(`Unknown preview target: ${name}`);515 if (!match) {
516 const valid = ALL_TARGETS.map(targetPackageName).join(", ");
517 throw new Error(`Unknown build target: ${name}. Expected one of: ${valid}`);
518 }
513 return match;519 return match;
514 });520 });
515 if (resolved.length === 0) throw new Error("EXECUTOR_PREVIEW_TARGETS resolved to an empty list");521 if (resolved.length === 0) throw new Error("Build target list resolved to empty");
516 return resolved;522 return resolved;
517};523};
518 524
@@ -919,6 +925,11 @@ const { values, positionals } = parseArgs({ · 400 unmodified lines
919 args: process.argv.slice(2),925 args: process.argv.slice(2),
920 options: {926 options: {
921 single: { type: "boolean", default: false },927 single: { type: "boolean", default: false },
928 // Build a specific target (or comma-separated set) by package name, e.g.
929 // `--target executor-windows-x64`. Used by the e2e VM harness to compile
930 // the guest's binary; without it `binary` builds the current platform
931 // (`--single`) or all targets.
932 target: { type: "string" },
922 mode: { type: "string", default: "production" },933 mode: { type: "string", default: "production" },
923 },934 },
924 allowPositionals: true,935 allowPositionals: true,
@@ -932,7 +943,11 @@ if (mode !== "production" && mode !== "development") { · 7 unmodified lines
932}943}
933 944
934if (command === "binary") {945if (command === "binary") {
935 const targets = values.single ? ALL_TARGETS.filter(isCurrentPlatform) : ALL_TARGETS;946 const targets = values.target
947 ? resolveTargetsFromEnv(values.target)
948 : values.single
949 ? ALL_TARGETS.filter(isCurrentPlatform)
950 : ALL_TARGETS;
936 const binaries = await buildBinaries(targets, mode);951 const binaries = await buildBinaries(targets, mode);
937 await buildWrapperPackage(binaries);952 await buildWrapperPackage(binaries);
938} else if (command === "preview") {953} else if (command === "preview") {
apps/cli/src/service.test.ts+2−2
@@ -50,7 +50,7 @@ describe("service unit generation", () => { · 49 unmodified lines
50 50
51 it("never leaks the auth password into the unit", () => {51 it("never leaks the auth password into the unit", () => {
52 const plist = generateLaunchdPlist(launchdInput);52 const plist = generateLaunchdPlist(launchdInput);
53 // The secret lives in the 0600 service.key, never in the plist env.53 // No secret in the unit — the daemon reads the bearer from auth.json at boot.
54 expect(plist).not.toContain("EXECUTOR_AUTH_PASSWORD");54 expect(plist).not.toContain("EXECUTOR_AUTH_PASSWORD");
55 });55 });
56 56
@@ -95,7 +95,7 @@ describe("service unit generation", () => { · 38 unmodified lines
95 '"C:\\Program Files\\Executor\\executor.exe" daemon run --foreground --port 4789',95 '"C:\\Program Files\\Executor\\executor.exe" daemon run --foreground --port 4789',
96 );96 );
97 expect(wrapper).toContain('1>> "C:\\Users\\x\\.executor\\logs\\daemon.log"');97 expect(wrapper).toContain('1>> "C:\\Users\\x\\.executor\\logs\\daemon.log"');
98 // The secret is never baked into the wrapper — the daemon reads service.key.98 // No secret in the wrapper — the daemon reads the bearer from auth.json at boot.
99 expect(wrapper).not.toContain("EXECUTOR_AUTH_PASSWORD");99 expect(wrapper).not.toContain("EXECUTOR_AUTH_PASSWORD");
100 });100 });
101 101
e2e/cli/service-lifecycle.test.ts+128−0
@@ -0,0 +1,128 @@
1// The supervised daemon's durability as a WATCHABLE terminal recording: register
2// a REAL integration, REBOOT the daemon's machine for real (the on-screen
3// spinner runs for the actual reboot), then show the integration still there.
4// Same assertions as restart-persistence — but filmed, so you can press play
5// instead of trusting a green check. Runs against the cli-* VM targets.
6import { randomBytes } from "node:crypto";
7import { join } from "node:path";
8
9import { expect } from "@effect/vitest";
10import { Effect } from "effect";
11
12import { composePluginApi } from "@executor-js/api/server";
13import { openApiHttpPlugin } from "@executor-js/plugin-openapi/api";
14
15import { withChatTheater } from "../src/clients/chat-theater";
16import { scenario } from "../src/scenario";
17import { Api, Cli, Restart, RunDir, Target } from "../src/services";
18
19const api = composePluginApi([openApiHttpPlugin()] as const);
20
21/** Inline OpenAPI 3 spec with a single GET /ping (its server is never called). */
22const pingSpec = JSON.stringify({
23 openapi: "3.0.3",
24 info: { title: "Reboot Lifecycle API", version: "1.0.0" },
25 servers: [{ url: "http://127.0.0.1:59998" }],
26 paths: {
27 "/ping": {
28 get: {
29 operationId: "getPing",
30 summary: "Liveness ping",
31 responses: { "200": { description: "pong" } },
32 },
33 },
34 },
35});
36
37scenario(
38 "Supervised daemon · an integration survives a real machine reboot (recorded)",
39 {},
40 Effect.gen(function* () {
41 const target = yield* Target;
42 const restart = yield* Restart;
43 const { client } = yield* Api;
44 const cli = yield* Cli;
45 const runDir = yield* RunDir;
46
47 const slug = `reboot-film-${randomBytes(4).toString("hex")}`;
48
49 yield* withChatTheater(
50 cli,
51 { title: "executor — supervised daemon", record: join(runDir, "terminal.cast") },
52 (chat) =>
53 Effect.gen(function* () {
54 yield* chat.user("If this machine reboots, do my connected integrations survive?");
55 yield* chat.assistant(
56 "Let's prove it — register a real integration, reboot the daemon's machine for real, then check it's still there.",
57 );
58
59 const before = yield* client(api, yield* target.newIdentity());
60
61 const added = yield* chat.tool(
62 {
63 name: "executor call executor.openapi.addSpec",
64 input: `slug: ${slug}\nspec: inline OpenAPI (GET /ping)`,
65 result: (a) => `registered — ${a.toolCount} tool(s)`,
66 },
67 before.openapi.addSpec({
68 payload: {
69 spec: { kind: "blob", value: pingSpec },
70 slug,
71 authenticationTemplate: [],
72 },
73 }),
74 );
75 expect(added.toolCount, "the spec registered with tools").toBeGreaterThan(0);
76
77 const listed = yield* chat.tool(
78 {
79 name: "executor tools sources",
80 result: (rows) =>
81 rows.map((r) => String(r.slug)).includes(slug) ? `${slug} is listed` : "NOT listed",
82 },
83 before.integrations.list(),
84 );
85 expect(
86 listed.map((i) => String(i.slug)),
87 "listed before the reboot",
88 ).toContain(slug);
89
90 // The spinner here runs for the ENTIRE real reboot — the supervised
91 // service must auto-start at boot for this to ever return.
92 yield* chat.tool(
93 {
94 name: "reboot the daemon's machine",
95 input: "guest OS reboot — the OS service manager must auto-start the daemon at boot",
96 result: () => "back online; daemon auto-started",
97 },
98 restart(),
99 );
100
101 const after = yield* client(api, yield* target.newIdentity());
102 yield* Effect.ensuring(
103 Effect.gen(function* () {
104 const survived = yield* chat.tool(
105 {
106 name: "executor tools sources",
107 result: (rows) =>
108 rows.map((r) => String(r.slug)).includes(slug)
109 ? `${slug} SURVIVED the reboot`
110 : "VANISHED",
111 },
112 after.integrations.list(),
113 );
114 expect(
115 survived.map((i) => String(i.slug)),
116 "survived the reboot",
117 ).toContain(slug);
118 yield* chat.assistant(
119 "It survived — the OS restarted the daemon at boot and its data was intact.",
120 );
121 }),
122 // Shared guest, but ephemeral; still, never leave the spec behind.
123 after.openapi.removeSpec({ params: { slug } }).pipe(Effect.ignore),
124 );
125 }),
126 );
127 }),
128);
e2e/desktop/supervised-attach.test.ts+182−0
@@ -0,0 +1,182 @@
1// Desktop-only, on camera: the app ATTACHES to an already-running OS-supervised
2// daemon instead of spawning its own sidecar. We start a real supervised gateway
3// (the desktop sidecar server in EXECUTOR_SUPERVISED mode → it self-publishes a
4// manifest of kind "cli-daemon"), launch the Electron app pointed at the same
5// HOME, and prove it attached: the manifest still names OUR daemon's pid (a
6// spawned sidecar would be a fresh pid + kind "desktop-sidecar"). The recording
7// (session.mp4 + screenshots) is the artifact; the waits are the assertions. No
8// launchd — only a throwaway home and one short-lived daemon process.
9import { type ChildProcess, execFile, spawn } from "node:child_process";
10import { existsSync, mkdtempSync, readFileSync, rmSync } from "node:fs";
11import { createRequire } from "node:module";
12import net from "node:net";
13import { tmpdir } from "node:os";
14import { join } from "node:path";
15import { fileURLToPath } from "node:url";
16import { promisify } from "node:util";
17
18import { expect } from "@effect/vitest";
19import { Effect } from "effect";
20import { _electron } from "playwright";
21
22import { scenario } from "../src/scenario";
23import { RunDir } from "../src/services";
24import { waitForHttp } from "../setup/boot";
25
26const appDir = fileURLToPath(new URL("../../apps/desktop/", import.meta.url));
27const repoRoot = fileURLToPath(new URL("../../", import.meta.url));
28const sidecarServer = join(appDir, "src/sidecar/server.ts");
29const clientDir = join(repoRoot, "apps/local/dist");
30const electronBinary = createRequire(join(appDir, "package.json"))("electron") as string;
31
32const freePort = (): Promise<number> =>
33 new Promise((resolve, reject) => {
34 const srv = net.createServer();
35 srv.on("error", reject);
36 srv.listen(0, "127.0.0.1", () => {
37 const port = (srv.address() as net.AddressInfo).port;
38 srv.close(() => resolve(port));
39 });
40 });
41
42interface Manifest {
43 readonly kind: string;
44 readonly pid: number;
45}
46
47interface DaemonStart {
48 readonly child: ChildProcess;
49 readonly ready: boolean;
50 readonly stderr: string;
51}
52
53/** Spawn the supervised gateway; resolves once it announces EXECUTOR_READY (or
54 * times out / exits early, with `ready: false`). The caller asserts readiness,
55 * so the executor only ever resolves. */
56const startSupervisedDaemon = (env: NodeJS.ProcessEnv): Promise<DaemonStart> =>
57 new Promise((resolve) => {
58 const child = spawn("bun", ["run", sidecarServer], {
59 cwd: repoRoot,
60 env,
61 stdio: ["ignore", "pipe", "pipe"],
62 });
63 let stderr = "";
64 const settle = (ready: boolean) => resolve({ child, ready, stderr });
65 const timer = setTimeout(() => settle(false), 60_000);
66 child.stdout.on("data", (chunk: Buffer) => {
67 if (chunk.toString().includes("EXECUTOR_READY:")) {
68 clearTimeout(timer);
69 settle(true);
70 }
71 });
72 child.stderr.on("data", (chunk: Buffer) => {
73 stderr += chunk.toString();
74 });
75 child.on("exit", () => {
76 clearTimeout(timer);
77 settle(false);
78 });
79 });
80
81scenario(
82 "Desktop · attaches to the OS-supervised daemon instead of spawning a sidecar",
83 { timeout: 240_000 },
84 Effect.gen(function* () {
85 const runDir = yield* RunDir;
86 yield* Effect.promise(() => run(runDir));
87 }),
88);
89
90const run = async (runDir: string) => {
91 const home = mkdtempSync(join(tmpdir(), "executor-attach-e2e-"));
92 const dataDir = join(home, ".executor");
93 const manifestPath = join(dataDir, "server-control", "server.json");
94 const videoTmp = join(runDir, ".video-tmp");
95 const port = await freePort();
96
97 let daemon: ChildProcess | undefined;
98 let app: Awaited<ReturnType<typeof _electron.launch>> | undefined;
99 let stepIndex = 0;
100
101 try {
102 const started = await startSupervisedDaemon({
103 ...process.env,
104 HOME: home,
105 EXECUTOR_SUPERVISED: "1",
106 EXECUTOR_DATA_DIR: dataDir,
107 EXECUTOR_PORT: String(port),
108 EXECUTOR_HOST: "127.0.0.1",
109 EXECUTOR_AUTH_TOKEN: "supervised-attach-film",
110 EXECUTOR_CLIENT_DIR: clientDir,
111 });
112 daemon = started.child;
113 expect(started.ready, `supervised daemon became ready; stderr:\n${started.stderr}`).toBe(true);
114 await waitForHttp(`http://127.0.0.1:${port}/`, { timeoutMs: 30_000 });
115
116 const daemonManifest = JSON.parse(readFileSync(manifestPath, "utf8")) as Manifest;
117 expect(daemonManifest.kind, "the running daemon advertises itself as cli-daemon").toBe(
118 "cli-daemon",
119 );
120 const daemonPid = daemonManifest.pid;
121
122 app = await _electron.launch({
123 executablePath: electronBinary,
124 args: [appDir],
125 cwd: appDir,
126 env: { ...process.env, HOME: home },
127 recordVideo: { dir: videoTmp, size: { width: 1280, height: 800 } },
128 timeout: 120_000,
129 });
130
131 const page = await app.firstWindow({ timeout: 120_000 });
132 const step = async (label: string, body: () => Promise<void>) => {
133 await body();
134 stepIndex += 1;
135 const slug = label.toLowerCase().replace(/[^a-z0-9]+/g, "-");
136 await page.screenshot({
137 path: join(runDir, `${String(stepIndex).padStart(2, "0")}-${slug}.png`),
138 });
139 };
140
141 // The window only loads the console once the app has a connection — and it
142 // attaches to the supervised daemon before it would ever spawn a sidecar.
143 await step("desktop boots into the console", async () => {
144 await page.getByText("Settings").first().waitFor({ timeout: 120_000 });
145 });
146
147 // The proof it ATTACHED rather than spawned: the manifest is untouched —
148 // same pid, still cli-daemon. A managed sidecar would have rewritten it to
149 // kind "desktop-sidecar" with a fresh child pid.
150 await step("server manifest still names the supervised daemon", async () => {
151 const after = JSON.parse(readFileSync(manifestPath, "utf8")) as Manifest;
152 expect(after.kind, "still the supervised daemon (not a desktop sidecar)").toBe("cli-daemon");
153 expect(after.pid, "the desktop attached to our daemon, not a new sidecar").toBe(daemonPid);
154 });
155 } finally {
156 const page = app?.windows()[0];
157 const video = page?.video();
158 await app?.close().catch(() => {});
159 const recordedPath = await video?.path().catch(() => undefined);
160 if (recordedPath && existsSync(recordedPath)) {
161 await promisify(execFile)("ffmpeg", [
162 "-y",
163 "-i",
164 recordedPath,
165 "-c:v",
166 "libx264",
167 "-preset",
168 "veryfast",
169 "-crf",
170 "26",
171 "-pix_fmt",
172 "yuv420p",
173 "-movflags",
174 "+faststart",
175 join(runDir, "session.mp4"),
176 ]).catch(() => {});
177 }
178 daemon?.kill("SIGTERM");
179 rmSync(videoTmp, { recursive: true, force: true });
180 rmSync(home, { recursive: true, force: true });
181 }
182};
e2e/setup/boot.ts+24−0
@@ -100,3 +100,27 @@ export const waitForHttp = async ( · 99 unmodified lines
100 }100 }
101 throw new Error(`timed out waiting for ${url}: ${String(lastError)}`);101 throw new Error(`timed out waiting for ${url}: ${String(lastError)}`);
102};102};
103
104/**
105 * Reboot DOWN-GATE: wait until `url` stops answering (fetch rejects — connection
106 * refused/reset). An orderly OS shutdown keeps the daemon serving for several
107 * seconds, and a reconnecting tunnel re-establishes the forward, so polling for
108 * "up" right after a reboot command can false-pass a reboot that never happened.
109 * Gating on the server actually going DOWN first makes restart-persistence prove
110 * a real reboot. Throws if it never goes down within the deadline.
111 */
112export const waitForHttpDown = async (
113 url: string,
114 options: { readonly timeoutMs?: number } = {},
115): Promise<void> => {
116 const deadline = Date.now() + (options.timeoutMs ?? 120_000);
117 while (Date.now() < deadline) {
118 try {
119 await fetch(url, { redirect: "manual", signal: AbortSignal.timeout(2000) });
120 } catch {
121 return; // the server (or its tunnel) is gone — the reboot took
122 }
123 await new Promise((resolve) => setTimeout(resolve, 500));
124 }
125 throw new Error(`${url} never became unreachable — the reboot may not have taken`);
126};
e2e/setup/cli-linux.globalsetup.ts+3−0
@@ -0,0 +1,3 @@
1import { setupCliTarget } from "./cli.globalsetup";
2
3export default (): Promise<(() => Promise<void>) | void> => setupCliTarget("linux");
e2e/setup/cli-macos.globalsetup.ts+3−0
@@ -0,0 +1,3 @@
1import { setupCliTarget } from "./cli.globalsetup";
2
3export default (): Promise<(() => Promise<void>) | void> => setupCliTarget("macos");
e2e/setup/cli-windows.globalsetup.ts+3−0
@@ -0,0 +1,3 @@
1import { setupCliTarget } from "./cli.globalsetup";
2
3export default (): Promise<(() => Promise<void>) | void> => setupCliTarget("windows");
e2e/setup/cli.globalsetup.ts+97−0
@@ -0,0 +1,97 @@
1// Boot the `cli` target: build the guest's `executor` binary, provision a VM,
2// install it as a supervised OS service, and forward its loopback HTTP port to
3// the host over a reconnecting SSH tunnel. Publishes connection + reboot info
4// via env (inherited by the test workers spawned afterward). Per-OS entrypoints
5// (cli-macos.globalsetup.ts, …) call setupCliTarget with their OS.
6//
7// macOS + Linux run on local tart guests (Apple-Silicon host); Windows runs on
8// an ephemeral EC2 instance. The supervised daemon is bearer-gated: it mints/
9// loads its token into auth.json, which we read from the guest and publish so
10// the api surface authenticates with `Authorization: Bearer`.
11import { buildGuestBinary } from "../src/vm/build-binary";
12import { ec2Vm } from "../src/vm/ec2";
13import { tartVm } from "../src/vm/tart";
14import type { VmArch, VmHandle, VmOs } from "../src/vm/types";
15import { waitForHttp } from "./boot";
16
17const PORT = 4789;
18
19// Per-OS guest specifics: working dir, binary name, and the shell idioms for
20// prep/cleanup — Unix `sh` on tart (macOS/Linux), PowerShell on EC2 Windows.
21const guestPlan = (os: VmOs) => {
22 if (os === "windows") {
23 const dir = "C:/ed";
24 const exe = `${dir}/executor.exe`;
25 return {
26 dir,
27 prep: `Remove-Item -Recurse -Force '${dir}' -ErrorAction SilentlyContinue; New-Item -ItemType Directory -Force -Path '${dir}' | Out-Null`,
28 postPush: "Write-Output ok", // no chmod/quarantine on Windows
29 install: `& '${exe}' service install --port ${PORT}`,
30 readToken: 'Get-Content "$env:USERPROFILE\\.executor\\server-control\\auth.json" -Raw',
31 uninstall: `& '${exe}' service uninstall`,
32 };
33 }
34 const dir = "~/ed";
35 const exe = `${dir}/executor`;
36 return {
37 dir,
38 prep: `rm -rf ${dir} && mkdir -p ${dir}`,
39 // macOS quarantines scp'd executables; clear it so the binary can run.
40 postPush:
41 os === "macos"
42 ? `chmod +x ${exe}; xattr -dr com.apple.quarantine ${dir} 2>/dev/null || true`
43 : `chmod +x ${exe}`,
44 install: `${exe} service install --port ${PORT}`,
45 readToken: "cat ~/.executor/server-control/auth.json",
46 uninstall: `${exe} service uninstall`,
47 };
48};
49
50export async function setupCliTarget(os: VmOs): Promise<(() => Promise<void>) | void> {
51 process.env.E2E_VM_OS = os; // so the worker-side target resolves the same OS
52
53 const arch: VmArch = os === "windows" ? "x64" : "arm64";
54 const binDir = await buildGuestBinary(os, arch);
55 const vm: VmHandle =
56 os === "windows" ? await ec2Vm(os, arch).provision() : await tartVm(os, arch).provision();
57 const plan = guestPlan(os);
58
59 let tunnelClose: (() => void) | undefined;
60 try {
61 await vm.ssh(plan.prep);
62 await vm.push(`${binDir}/.`, os === "windows" ? plan.dir : `${plan.dir}/`);
63 await vm.ssh(plan.postPush);
64
65 const install = await vm.ssh(plan.install);
66 if (install.code !== 0) {
67 throw new Error(`service install failed: ${install.stderr.trim() || install.stdout.trim()}`);
68 }
69
70 // The supervised daemon mints/loads its bearer into auth.json on first boot.
71 const tokenRaw = (await vm.ssh(plan.readToken)).stdout.trim();
72 const token = (JSON.parse(tokenRaw) as { token: string }).token;
73
74 const tunnel = await vm.tunnel(PORT);
75 tunnelClose = tunnel.close;
76 const baseUrl = `http://127.0.0.1:${tunnel.localPort}`;
77 await waitForHttp(`${baseUrl}/`, { timeoutMs: 60_000 });
78
79 process.env.E2E_CLI_BASE_URL = baseUrl;
80 process.env.E2E_CLI_AUTH_TOKEN = token;
81 process.env.E2E_CLI_VM_HOST = vm.host;
82 if (vm.sshKeyPath) process.env.E2E_CLI_SSH_KEY = vm.sshKeyPath;
83 process.env.E2E_CLI_TUNNEL_PORT = String(tunnel.localPort);
84 process.env.E2E_CLI_BIN_DIR = plan.dir;
85 } catch (error) {
86 tunnelClose?.();
87 await vm.ssh(plan.uninstall).catch(() => undefined);
88 await vm.discard();
89 throw error;
90 }
91
92 return async () => {
93 tunnelClose?.();
94 await vm.ssh(plan.uninstall).catch(() => undefined);
95 await vm.discard();
96 };
97}
e2e/src/vm/build-binary.ts+35−0
@@ -0,0 +1,35 @@
1// Build the compiled `executor` for a guest os/arch. `service install` refuses
2// to run from a dev (.ts) entrypoint, so the VM targets need a real binary —
3// produced via the `--target` flag on apps/cli/src/build.ts.
4
5import { execFile } from "node:child_process";
6import { existsSync } from "node:fs";
7import path from "node:path";
8import { promisify } from "node:util";
9
10import type { VmArch, VmOs } from "./types";
11
12const execFileP = promisify(execFile);
13
14const PLATFORM_TAG: Record<VmOs, string> = { macos: "darwin", linux: "linux", windows: "windows" };
15
16// e2e/src/vm/build-binary.ts → repo root.
17const REPO_ROOT = path.resolve(import.meta.dirname, "../../..");
18const CLI_DIR = path.join(REPO_ROOT, "apps", "cli");
19
20/**
21 * Build the `executor` binary for a guest and return its `bin` directory
22 * (executor[.exe] + the native libsql/keyring modules). The build cleans
23 * `dist/` each run, so callers should push the result before building another.
24 */
25export const buildGuestBinary = async (os: VmOs, arch: VmArch): Promise<string> => {
26 const target = `executor-${PLATFORM_TAG[os]}-${arch}`;
27 await execFileP("bun", ["run", "src/build.ts", "binary", "--target", target], {
28 cwd: CLI_DIR,
29 maxBuffer: 256 * 1024 * 1024,
30 });
31 const binDir = path.join(CLI_DIR, "dist", target, "bin");
32 const exe = path.join(binDir, os === "windows" ? "executor.exe" : "executor");
33 if (!existsSync(exe)) throw new Error(`buildGuestBinary: ${exe} not produced`);
34 return binDir;
35};
e2e/src/vm/ec2.ts+422−0
@@ -0,0 +1,422 @@
1// ec2 provider: ephemeral guests on AWS EC2 for the cross-OS supervised-daemon
2// e2e where tart can't run (Windows; optionally Linux). Mirrors tart.ts — launch
3// a fresh instance, drive over SSH (key-based; PowerShell on Windows), REBOOT for
4// real, tear down.
5//
6// Credentials are NEVER embedded here: the `aws` CLI uses the ambient sign-in
7// (`aws configure` / env). Every instance is tagged `executor-e2e` and always
8// terminated on discard; the security group is scoped to this host's egress IP.
9//
10// Reboot is gated on a real boot-time change (Windows `LastBootUpTime`), not mere
11// SSH reachability — an orderly shutdown keeps the daemon serving for several
12// seconds, so "SSH answered" alone can false-pass a reboot that never happened.
13
14import { execFile } from "node:child_process";
15import { chmodSync, mkdtempSync, writeFileSync } from "node:fs";
16import net from "node:net";
17import { tmpdir } from "node:os";
18import { join } from "node:path";
19import { promisify } from "node:util";
20
21import {
22 type SshResult,
23 sleep,
24 type VmArch,
25 type VmHandle,
26 type VmOs,
27 type VmProvider,
28} from "./types";
29
30const execFileP = promisify(execFile);
31
32const REGION = process.env.E2E_EC2_REGION ?? "us-west-2";
33const INSTANCE_TYPE = process.env.E2E_EC2_INSTANCE_TYPE ?? "t3.medium";
34const TAG = "executor-e2e";
35
36const SSH_OPTS = [
37 "-o",
38 "StrictHostKeyChecking=no",
39 "-o",
40 "UserKnownHostsFile=/dev/null",
41 "-o",
42 "ConnectTimeout=10",
43 "-o",
44 "ServerAliveInterval=10",
45 "-o",
46 "LogLevel=ERROR",
47];
48
49const guestUser = (os: VmOs): string =>
50 os === "windows" ? "Administrator" : (process.env.E2E_EC2_LINUX_USER ?? "ubuntu");
51
52/**
53 * Reboot an EC2 guest by address, statelessly (no live handle) — the mirror of
54 * tart's sshRebootGuest, for the worker-side `restart()`. The connection drops
55 * mid-call, so errors are swallowed; the caller's down-gate + up-poll confirm
56 * the real reboot.
57 */
58export const ec2RebootGuest = async (
59 host: string,
60 keyPath: string,
61 os: VmOs = "windows",
62): Promise<void> => {
63 const cmd = os === "windows" ? "Restart-Computer -Force" : "sudo reboot";
64 await execFileP("ssh", ["-i", keyPath, ...SSH_OPTS, `${guestUser(os)}@${host}`, cmd]).catch(
65 () => undefined,
66 );
67};
68
69const aws = async (args: ReadonlyArray<string>): Promise<string> => {
70 const { stdout } = await execFileP("aws", ["--region", REGION, "--output", "text", ...args], {
71 maxBuffer: 64 * 1024 * 1024,
72 });
73 return stdout.trim();
74};
75
76/** This host's public egress IP, for the inbound-SSH security-group rule. */
77const egressIp = async (): Promise<string> => {
78 const { stdout } = await execFileP("curl", [
79 "-s",
80 "--max-time",
81 "10",
82 "https://checkip.amazonaws.com",
83 ]);
84 return stdout.trim();
85};
86
87/** Latest AWS-published base AMI for the guest OS (resolve dynamically — ids rotate). */
88const latestAmi = async (os: VmOs): Promise<string> => {
89 if (os === "windows") {
90 const viaSsm = await aws([
91 "ssm",
92 "get-parameters",
93 "--names",
94 "/aws/service/ami-windows-latest/Windows_Server-2022-English-Full-Base",
95 "--query",
96 "Parameters[0].Value",
97 ]).catch(() => "");
98 if (viaSsm && viaSsm !== "None") return viaSsm;
99 return aws([
100 "ec2",
101 "describe-images",
102 "--owners",
103 "amazon",
104 "--filters",
105 "Name=name,Values=Windows_Server-2022-English-Full-Base-*",
106 "Name=state,Values=available",
107 "--query",
108 "reverse(sort_by(Images,&CreationDate))[0].ImageId",
109 ]);
110 }
111 return aws([
112 "ssm",
113 "get-parameters",
114 "--names",
115 "/aws/service/canonical/ubuntu/server/22.04/stable/current/amd64/hvm/ebs-gp2/ami-id",
116 "--query",
117 "Parameters[0].Value",
118 ]);
119};
120
121const defaultSubnet = async (): Promise<string> => {
122 const vpc = await aws([
123 "ec2",
124 "describe-vpcs",
125 "--filters",
126 "Name=isDefault,Values=true",
127 "--query",
128 "Vpcs[0].VpcId",
129 ]);
130 const subnet = await aws([
131 "ec2",
132 "describe-subnets",
133 "--filters",
134 `Name=vpc-id,Values=${vpc}`,
135 "Name=default-for-az,Values=true",
136 "--query",
137 "Subnets[0].SubnetId",
138 ]);
139 return subnet && subnet !== "None"
140 ? subnet
141 : aws([
142 "ec2",
143 "describe-subnets",
144 "--filters",
145 `Name=vpc-id,Values=${vpc}`,
146 "--query",
147 "Subnets[0].SubnetId",
148 ]);
149};
150
151/** Create (idempotently) a security group allowing inbound SSH from this host. */
152const ensureSecurityGroup = async (myIp: string): Promise<string> => {
153 const name = `${TAG}-sg`;
154 let sg = await aws([
155 "ec2",
156 "describe-security-groups",
157 "--filters",
158 `Name=group-name,Values=${name}`,
159 "--query",
160 "SecurityGroups[0].GroupId",
161 ]).catch(() => "");
162 if (!sg || sg === "None") {
163 sg = await aws([
164 "ec2",
165 "create-security-group",
166 "--group-name",
167 name,
168 "--description",
169 "executor e2e ephemeral guests (SSH from CI host)",
170 "--query",
171 "GroupId",
172 ]);
173 }
174 // Authorize this host's IP for SSH; ignore "already exists".
175 await aws([
176 "ec2",
177 "authorize-security-group-ingress",
178 "--group-id",
179 sg,
180 "--protocol",
181 "tcp",
182 "--port",
183 "22",
184 "--cidr",
185 `${myIp}/32`,
186 ]).catch(() => undefined);
187 return sg;
188};
189
190/** PowerShell user-data: enable OpenSSH, default the shell to PowerShell, and
191 * authorize our public key for the Administrator account. */
192const windowsUserData = (publicKey: string): string =>
193 [
194 "<powershell>",
195 "Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0",
196 "Set-Service -Name sshd -StartupType Automatic",
197 "Start-Service sshd",
198 "New-ItemProperty -Path 'HKLM:\\SOFTWARE\\OpenSSH' -Name DefaultShell -Value 'C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe' -PropertyType String -Force",
199 "New-NetFirewallRule -Name sshd -DisplayName 'OpenSSH Server (sshd)' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 22 -ErrorAction SilentlyContinue",
200 "$akf = 'C:\\ProgramData\\ssh\\administrators_authorized_keys'",
201 `Set-Content -Path $akf -Value '${publicKey}'`,
202 "icacls $akf /inheritance:r /grant 'Administrators:F' /grant 'SYSTEM:F'",
203 "</powershell>",
204 ].join("\n");
205
206const linuxUserData = (publicKey: string): string =>
207 ["#cloud-config", "ssh_authorized_keys:", ` - ${publicKey}`].join("\n");
208
209const freePort = (): Promise<number> =>
210 new Promise((resolve, reject) => {
211 const srv = net.createServer();
212 srv.on("error", reject);
213 srv.listen(0, "127.0.0.1", () => {
214 const port = (srv.address() as net.AddressInfo).port;
215 srv.close(() => resolve(port));
216 });
217 });
218
219const waitLocalPort = async (port: number, attempts = 40): Promise<void> => {
220 for (let i = 0; i < attempts; i++) {
221 const ok = await new Promise<boolean>((resolve) => {
222 const sock = net.connect({ host: "127.0.0.1", port }, () => {
223 sock.destroy();
224 resolve(true);
225 });
226 sock.on("error", () => resolve(false));
227 sock.setTimeout(1000, () => {
228 sock.destroy();
229 resolve(false);
230 });
231 });
232 if (ok) return;
233 await sleep(500);
234 }
235 throw new Error(`tunnel local port ${port} never came up`);
236};
237
238export const ec2Vm = (os: VmOs, arch: VmArch = "x64"): VmProvider => ({
239 os,
240 provision: async () => {
241 const user = guestUser(os);
242 // A throwaway SSH keypair, authorized via user-data (no EC2 key pair needed —
243 // we drive over OpenSSH key auth, not the Windows password).
244 const keyDir = mkdtempSync(join(tmpdir(), "executor-ec2-"));
245 const keyPath = join(keyDir, "id");
246 await execFileP("ssh-keygen", ["-t", "ed25519", "-N", "", "-q", "-f", keyPath]);
247 chmodSync(keyPath, 0o600);
248 const publicKey = (await execFileP("ssh-keygen", ["-y", "-f", keyPath])).stdout.trim();
249
250 const [myIp, ami, subnet] = await Promise.all([egressIp(), latestAmi(os), defaultSubnet()]);
251 const sg = await ensureSecurityGroup(myIp);
252 const userData = os === "windows" ? windowsUserData(publicKey) : linuxUserData(publicKey);
253 const userDataFile = join(keyDir, "user-data.txt");
254 writeFileSync(userDataFile, userData);
255
256 const instanceId = await aws([
257 "ec2",
258 "run-instances",
259 "--image-id",
260 ami,
261 "--instance-type",
262 INSTANCE_TYPE,
263 "--count",
264 "1",
265 "--security-group-ids",
266 sg,
267 "--subnet-id",
268 subnet,
269 "--associate-public-ip-address",
270 "--instance-initiated-shutdown-behavior",
271 "terminate",
272 "--user-data",
273 `file://${userDataFile}`,
274 "--tag-specifications",
275 `ResourceType=instance,Tags=[{Key=Name,Value=${TAG}-${os}},{Key=purpose,Value=e2e}]`,
276 "--query",
277 "Instances[0].InstanceId",
278 ]);
279
280 let ip = "";
281 const tunnelClosers: Array<() => void> = [];
282
283 const ssh = async (command: string): Promise<SshResult> => {
284 try {
285 const { stdout, stderr } = await execFileP(
286 "ssh",
287 ["-i", keyPath, ...SSH_OPTS, `${user}@${ip}`, command],
288 { maxBuffer: 64 * 1024 * 1024 },
289 );
290 return { stdout, stderr, code: 0 };
291 } catch (err) {
292 const e = err as { stdout?: string; stderr?: string; code?: number };
293 return {
294 stdout: e.stdout ?? "",
295 stderr: e.stderr ?? "",
296 code: typeof e.code === "number" ? e.code : 1,
297 };
298 }
299 };
300
301 const waitSshUp = async (attempts: number): Promise<boolean> => {
302 for (let i = 0; i < attempts; i++) {
303 if ((await ssh(os === "windows" ? "echo ok" : "true")).code === 0) return true;
304 await sleep(5000);
305 }
306 return false;
307 };
308
309 const waitSshDown = async (attempts = 40): Promise<void> => {
310 for (let i = 0; i < attempts; i++) {
311 if ((await ssh("echo up").catch(() => ({ code: 1 }) as SshResult)).code !== 0) return;
312 await sleep(3000);
313 }
314 // never observed down — caller's boot-time check is the backstop.
315 };
316
317 const bootTime = async (): Promise<string> =>
318 os === "windows"
319 ? (
320 await ssh("(Get-CimInstance Win32_OperatingSystem).LastBootUpTime.ToString('o')")
321 ).stdout.trim()
322 : (await ssh("cat /proc/sys/kernel/random/boot_id")).stdout.trim();
323
324 const handle: VmHandle = {
325 os,
326 arch,
327 sshKeyPath: keyPath,
328 get host() {
329 return ip;
330 },
331 ssh,
332 push: async (localPath, remotePath) => {
333 await execFileP("scp", [
334 "-i",
335 keyPath,
336 "-r",
337 ...SSH_OPTS,
338 localPath,
339 `${user}@${ip}:${remotePath}`,
340 ]);
341 },
342 reboot: async () => {
343 const before = await bootTime();
344 await ssh(os === "windows" ? "Restart-Computer -Force" : "sudo reboot").catch(
345 () => undefined,
346 );
347 await waitSshDown();
348 if (!(await waitSshUp(60))) throw new Error(`ec2 ${os}: SSH did not return after reboot`);
349 const after = await bootTime();
350 if (before && after && before === after) {
351 throw new Error(
352 `ec2 ${os}: boot time unchanged after reboot — the guest never actually rebooted`,
353 );
354 }
355 },
356 tunnel: async (guestPort) => {
357 const localPort = await freePort();
358 let closed = false;
359 let child: ReturnType<typeof import("node:child_process").spawn> | undefined;
360 const { spawn } = await import("node:child_process");
361 const spawnOnce = (): void => {
362 child = spawn(
363 "ssh",
364 [
365 "-i",
366 keyPath,
367 ...SSH_OPTS,
368 "-N",
369 "-L",
370 `${localPort}:127.0.0.1:${guestPort}`,
371 `${user}@${ip}`,
372 ],
373 { stdio: "ignore" },
374 );
375 child.on("exit", () => {
376 if (!closed) setTimeout(spawnOnce, 2000);
377 });
378 };
379 spawnOnce();
380 const close = (): void => {
381 closed = true;
382 child?.kill();
383 };
384 tunnelClosers.push(close);
385 await waitLocalPort(localPort);
386 return { localPort, close };
387 },
388 discard: async () => {
389 for (const close of tunnelClosers) close();
390 await aws(["ec2", "terminate-instances", "--instance-ids", instanceId]).catch(
391 () => undefined,
392 );
393 },
394 };
395
396 // Wait for a public IP, then for OpenSSH (Windows boot + FoD install ≈ 2-4 min).
397 for (let i = 0; i < 60; i++) {
398 const got = await aws([
399 "ec2",
400 "describe-instances",
401 "--instance-ids",
402 instanceId,
403 "--query",
404 "Reservations[0].Instances[0].PublicIpAddress",
405 ]).catch(() => "");
406 if (got && got !== "None") {
407 ip = got;
408 break;
409 }
410 await sleep(5000);
411 }
412 if (!ip) {
413 await handle.discard();
414 throw new Error(`ec2 ${os}: no public IP within 300s`);
415 }
416 if (!(await waitSshUp(60))) {
417 await handle.discard();
418 throw new Error(`ec2 ${os}: SSH never came up`);
419 }
420 return handle;
421 },
422});
e2e/src/vm/tart.ts+224−0
@@ -0,0 +1,224 @@
1// tart provider: macOS + Linux guests on an Apple-Silicon host (the Mini).
2// Mirrors the by-hand reboot harness — clone a base image, boot headless, drive
3// over sshpass, reboot the guest OS for real, tear down the clone.
4
5import { execFile, spawn, type ChildProcess } from "node:child_process";
6import net from "node:net";
7import { promisify } from "node:util";
8
9import {
10 type SshResult,
11 sleep,
12 type Tunnel,
13 type VmArch,
14 type VmHandle,
15 type VmProvider,
16} from "./types";
17
18const execFileP = promisify(execFile);
19
20const TART = process.env.E2E_TART_BIN ?? "/opt/homebrew/bin/tart";
21const SSHPASS = process.env.E2E_SSHPASS_BIN ?? "/opt/homebrew/bin/sshpass";
22const SSH_OPTS = [
23 "-o",
24 "StrictHostKeyChecking=no",
25 "-o",
26 "UserKnownHostsFile=/dev/null",
27 "-o",
28 "ConnectTimeout=8",
29 "-o",
30 "ServerAliveInterval=5",
31 "-o",
32 "LogLevel=ERROR",
33];
34const GUEST_USER = "admin";
35const GUEST_PASS = "admin";
36
37/**
38 * Reboot a tart guest by address, with no live handle. `restart()` runs in a
39 * vitest worker (separate process from the globalsetup that owns the VM), so it
40 * re-derives the guest address from env and triggers the reboot statelessly —
41 * the reconnecting tunnel and a health poll confirm recovery.
42 */
43export const sshRebootGuest = async (ip: string): Promise<void> => {
44 await execFileP(SSHPASS, [
45 "-p",
46 GUEST_PASS,
47 "ssh",
48 ...SSH_OPTS,
49 `${GUEST_USER}@${ip}`,
50 "sudo reboot",
51 ]).catch(() => undefined); // the connection drops mid-call
52};
53
54const baseImage = (os: "macos" | "linux"): string =>
55 os === "macos"
56 ? (process.env.E2E_TART_MACOS_BASE ?? "executor-macos-base")
57 : (process.env.E2E_TART_LINUX_BASE ?? "executor-linux-base");
58
59/** Ask the OS for a free localhost port (for SSH tunnels). */
60const freePort = (): Promise<number> =>
61 new Promise((resolve, reject) => {
62 const srv = net.createServer();
63 srv.on("error", reject);
64 srv.listen(0, "127.0.0.1", () => {
65 const port = (srv.address() as net.AddressInfo).port;
66 srv.close(() => resolve(port));
67 });
68 });
69
70/** Resolve once a TCP connect to localhost:port succeeds (SSH bound the forward). */
71const waitLocalPort = async (port: number, attempts = 40): Promise<void> => {
72 for (let i = 0; i < attempts; i++) {
73 const ok = await new Promise<boolean>((resolve) => {
74 const sock = net.connect({ host: "127.0.0.1", port }, () => {
75 sock.destroy();
76 resolve(true);
77 });
78 sock.on("error", () => resolve(false));
79 sock.setTimeout(1000, () => {
80 sock.destroy();
81 resolve(false);
82 });
83 });
84 if (ok) return;
85 await sleep(500);
86 }
87 throw new Error(`tunnel local port ${port} never came up`);
88};
89
90export const tartVm = (os: "macos" | "linux", arch: VmArch = "arm64"): VmProvider => ({
91 os,
92 provision: async () => {
93 const name = `executor-e2e-${os}-${process.pid}-${Math.floor(performance.now())}`;
94 await execFileP(TART, ["clone", baseImage(os), name]);
95 const runProc = spawn(TART, ["run", name, "--no-graphics"], { stdio: "ignore" });
96
97 const tunnelClosers: Array<() => void> = [];
98 let ip = "";
99
100 const fetchIp = async (): Promise<boolean> => {
101 for (let i = 0; i < 90; i++) {
102 try {
103 const { stdout } = await execFileP(TART, ["ip", name]);
104 if (stdout.trim()) {
105 ip = stdout.trim();
106 return true;
107 }
108 } catch {
109 /* not booted yet */
110 }
111 await sleep(2000);
112 }
113 return false;
114 };
115
116 // Linux systemctl --user calls need XDG_RUNTIME_DIR; harmless elsewhere.
117 const wrap = (command: string): string =>
118 os === "linux" ? `export XDG_RUNTIME_DIR=/run/user/$(id -u); ${command}` : command;
119
120 const ssh = async (command: string): Promise<SshResult> => {
121 try {
122 const { stdout, stderr } = await execFileP(
123 SSHPASS,
124 ["-p", GUEST_PASS, "ssh", ...SSH_OPTS, `${GUEST_USER}@${ip}`, wrap(command)],
125 { maxBuffer: 32 * 1024 * 1024 },
126 );
127 return { stdout, stderr, code: 0 };
128 } catch (err) {
129 const e = err as { stdout?: string; stderr?: string; code?: number };
130 return {
131 stdout: e.stdout ?? "",
132 stderr: e.stderr ?? "",
133 code: typeof e.code === "number" ? e.code : 1,
134 };
135 }
136 };
137
138 const waitSsh = async (attempts: number): Promise<boolean> => {
139 for (let i = 0; i < attempts; i++) {
140 if ((await ssh("true")).code === 0) return true;
141 await sleep(2000);
142 }
143 return false;
144 };
145
146 const handle: VmHandle = {
147 os,
148 arch,
149 get host() {
150 return ip;
151 },
152 ssh,
153 push: async (localPath, remotePath) => {
154 await execFileP(SSHPASS, [
155 "-p",
156 GUEST_PASS,
157 "scp",
158 "-r",
159 ...SSH_OPTS,
160 localPath,
161 `${GUEST_USER}@${ip}:${remotePath}`,
162 ]);
163 },
164 reboot: async () => {
165 await ssh("sudo reboot").catch(() => undefined); // connection drops mid-call
166 await sleep(5000);
167 if (!(await fetchIp())) throw new Error(`tart ${os}: no IP after reboot`);
168 if (!(await waitSsh(120))) throw new Error(`tart ${os}: SSH did not return after reboot`);
169 },
170 tunnel: async (guestPort) => {
171 const localPort = await freePort();
172 // Reconnecting forward: when the guest reboots the ssh exits, so respawn
173 // it until closed. `restart()` health-polls through this local port, so
174 // it only goes green once the daemon AND the forward are back.
175 let closed = false;
176 let child: ChildProcess | undefined;
177 const spawnOnce = (): void => {
178 child = spawn(
179 SSHPASS,
180 [
181 "-p",
182 GUEST_PASS,
183 "ssh",
184 ...SSH_OPTS,
185 "-N",
186 "-L",
187 `${localPort}:127.0.0.1:${guestPort}`,
188 `${GUEST_USER}@${ip}`,
189 ],
190 { stdio: "ignore" },
191 );
192 child.on("exit", () => {
193 if (!closed) setTimeout(spawnOnce, 2000);
194 });
195 };
196 spawnOnce();
197 const close = (): void => {
198 closed = true;
199 child?.kill();
200 };
201 tunnelClosers.push(close);
202 await waitLocalPort(localPort);
203 const tunnel: Tunnel = { localPort, close };
204 return tunnel;
205 },
206 discard: async () => {
207 for (const close of tunnelClosers) close();
208 runProc.kill();
209 await sleep(1500);
210 await execFileP(TART, ["delete", name]).catch(() => undefined);
211 },
212 };
213
214 if (!(await fetchIp())) {
215 await handle.discard();
216 throw new Error(`tart ${os}: no IP within 180s`);
217 }
218 if (!(await waitSsh(90))) {
219 await handle.discard();
220 throw new Error(`tart ${os}: SSH never came up`);
221 }
222 return handle;
223 },
224});
e2e/src/vm/types.ts+51−0
@@ -0,0 +1,51 @@
1// VM substrate for the cross-OS supervised-daemon e2e targets.
2//
3// A VmHandle is a booted guest we can drive over SSH, REBOOT for real, and tear
4// down. Providers: tart (macOS + Linux, local on an Apple-Silicon host) and ec2
5// (Windows, ephemeral). This is the codified form of the by-hand reboot harness.
6
7export type VmOs = "macos" | "linux" | "windows";
8export type VmArch = "arm64" | "x64";
9
10export interface SshResult {
11 readonly stdout: string;
12 readonly stderr: string;
13 readonly code: number;
14}
15
16/** An open SSH local-forward (`localhost:localPort` → guest:guestPort). */
17export interface Tunnel {
18 readonly localPort: number;
19 close(): void;
20}
21
22export interface VmHandle {
23 readonly os: VmOs;
24 readonly arch: VmArch;
25 /** Current reachable address of the guest (re-resolved across reboots). */
26 readonly host: string;
27 /**
28 * Path to the SSH private key for key-based providers (EC2). Undefined for
29 * password-based providers (tart/sshpass). Published by globalsetup so the
30 * stateless worker-side `restart()` can reboot the guest.
31 */
32 readonly sshKeyPath?: string;
33 /** Run a command in the guest over SSH (shell on Unix, PowerShell on Windows). */
34 ssh(command: string): Promise<SshResult>;
35 /** Copy a local file or directory into the guest (recursive for directories). */
36 push(localPath: string, remotePath: string): Promise<void>;
37 /** Reboot the guest OS; resolves only once SSH is reachable again. */
38 reboot(): Promise<void>;
39 /** Forward `localhost:<localPort>` → `guest:<guestPort>` over SSH. */
40 tunnel(guestPort: number): Promise<Tunnel>;
41 /** Discard the VM (delete the tart clone / terminate the EC2 instance). */
42 discard(): Promise<void>;
43}
44
45export interface VmProvider {
46 readonly os: VmOs;
47 /** Boot a fresh guest and wait until SSH answers. */
48 provision(): Promise<VmHandle>;
49}
50
51export const sleep = (ms: number): Promise<void> => new Promise((r) => setTimeout(r, ms));
e2e/targets/cli.ts+59−0
@@ -0,0 +1,59 @@
1// The supervised local CLI daemon as a target. `executor service install` boots
2// an OS-managed daemon (launchd / systemd / Task Scheduler) inside a guest VM;
3// `restart()` reboots that guest for REAL, so restart-persistence proves the
4// boot-time auto-start path (RunAtLoad / linger / AtStartup), not just a process
5// restart. The guest daemon binds loopback, so globalsetup forwards it to a
6// local port over a reconnecting SSH tunnel — making target.baseUrl work
7// unchanged for the api surface. Boot + tunnel live in setup/cli.globalsetup.ts;
8// this target only reads what that published via env.
9import { Effect } from "effect";
10
11import { waitForHttp, waitForHttpDown } from "../setup/boot";
12import { ec2RebootGuest } from "../src/vm/ec2";
13import { sshRebootGuest } from "../src/vm/tart";
14import type { VmOs } from "../src/vm/types";
15import type { Capability, Identity, Target } from "../src/target";
16
17const env = (key: string): string => {
18 const value = process.env[key];
19 if (!value) throw new Error(`cli target: ${key} not set — did cli.globalsetup run?`);
20 return value;
21};
22
23export const cliTarget = (): Target => {
24 const baseUrl = env("E2E_CLI_BASE_URL");
25 const os = (process.env.E2E_VM_OS ?? "macos") as VmOs;
26
27 return {
28 name: process.env.E2E_TARGET ?? "cli",
29 baseUrl,
30 mcpUrl: `${baseUrl}/mcp`,
31 capabilities: new Set<Capability>(["api"]),
32 // The supervised daemon is bearer-gated (auth.json); globalsetup reads the
33 // token from the guest and publishes it. A wrong/absent token still gets a
34 // clean 401, which the api surface treats as "up".
35 newIdentity: () =>
36 Effect.sync(
37 (): Identity => ({
38 label: "cli-daemon",
39 headers: { Authorization: `Bearer ${env("E2E_CLI_AUTH_TOKEN")}` },
40 }),
41 ),
42 // A genuine machine reboot, not a service kick: reboot the guest OS, GATE on
43 // the daemon actually going down (an orderly shutdown serves for several
44 // seconds + the reconnecting tunnel re-forwards, so "reachable" right after
45 // the reboot command would false-pass), then wait for the supervised daemon
46 // to auto-start and serve again — proving the boot-time auto-start path.
47 restart: () =>
48 Effect.promise(async () => {
49 const host = env("E2E_CLI_VM_HOST");
50 if (os === "windows") {
51 await ec2RebootGuest(host, env("E2E_CLI_SSH_KEY"), os);
52 } else {
53 await sshRebootGuest(host);
54 }
55 await waitForHttpDown(`${baseUrl}/`, { timeoutMs: 120_000 });
56 await waitForHttp(`${baseUrl}/`, { timeoutMs: 240_000 });
57 }),
58 };
59};
e2e/targets/registry.ts+6−0
@@ -2,6 +2,7 @@ · 1 unmodified line
2// once per worker. Adding a target = one factory entry here + a project in2// once per worker. Adding a target = one factory entry here + a project in
3// vitest.config.ts + a globalsetup that boots (or attaches to) the instance.3// vitest.config.ts + a globalsetup that boots (or attaches to) the instance.
4import type { Target } from "../src/target";4import type { Target } from "../src/target";
5import { cliTarget } from "./cli";
5import { cloudTarget } from "./cloud";6import { cloudTarget } from "./cloud";
6import { cloudflareTarget } from "./cloudflare";7import { cloudflareTarget } from "./cloudflare";
7import { desktopTarget } from "./desktop";8import { desktopTarget } from "./desktop";
@@ -16,6 +17,11 @@ const factories: Record<string, () => Target> = { · 8 unmodified lines
16 cloudflare: cloudflareTarget,17 cloudflare: cloudflareTarget,
17 desktop: desktopTarget,18 desktop: desktopTarget,
18 local: localTarget,19 local: localTarget,
20 // The supervised CLI daemon inside a VM, one project per guest OS — restart()
21 // is a real reboot. See setup/cli.globalsetup.ts.
22 "cli-macos": cliTarget,
23 "cli-linux": cliTarget,
24 "cli-windows": cliTarget,
19};25};
20 26
21let current: Target | undefined;27let current: Target | undefined;
e2e/vitest.config.ts+15−0
@@ -68,6 +68,21 @@ export default defineConfig({ · 67 unmodified lines
68 fileParallelism: true,68 fileParallelism: true,
69 testTimeout: 180_000,69 testTimeout: 180_000,
70 }),70 }),
71 // The supervised CLI daemon inside a guest VM, one project per OS. The
72 // globalsetup provisions a VM, `executor service install`s the daemon, and
73 // tunnels it; restart() reboots the guest for REAL, so restart-persistence
74 // proves the boot-time auto-start path. Needs tart (macOS/Linux) or an EC2
75 // credential (Windows); not part of the default `npm run test` chain — run
76 // with `vitest run --project cli-macos` (etc.) on the Mini.
77 ...(["macos", "linux", "windows"] as const).map((os) =>
78 project(`cli-${os}`, {
79 include: ["scenarios/restart-persistence.test.ts", "cli/**/*.test.ts"],
80 env: { E2E_TARGET: `cli-${os}`, E2E_VM_OS: os },
81 fileParallelism: false,
82 testTimeout: 300_000,
83 hookTimeout: 900_000,
84 }),
85 ),
71 ],86 ],
72 },87 },
73});88});

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t