Trim agent instructions

Rhys Sullivan committed 3 months ago

Commit a3c0b0a1 parent 2226e723 files · +365 −349Browse files

Unified · Split

3 files

AGENTS.md +40 −69
e2e/AGENTS.md +42 −280
e2e/RUNNING.md +283 −0

@@ -1,84 +1,55 @@
1	# AGENTS.md	1	# Executor policy
2		2
3	Principles: the contracts that stay true while implementations churn. For how to	3	Stable contracts live here. Current setup and server mechanics are in
4	run, boot, share, or navigate things today, see [RUNNING.md](RUNNING.md); for	4	[RUNNING.md](RUNNING.md); e2e invariants are in
5	e2e scenarios, [e2e/AGENTS.md](e2e/AGENTS.md). Run `bun run bootstrap` first in	5	[e2e/AGENTS.md](e2e/AGENTS.md). Run `bun run bootstrap` in every fresh rift.
6	any fresh checkout or worktree.
7		6
8	## Task Completion	7	## Verification and evidence
9		8
10	- Tests use Effect Vitest. Run scoped tests with `vitest run ...`. `bun run test`	9	- Tests use Effect Vitest. Run scoped tests with `vitest run ...` or the
11	is fine (it delegates to Vitest); NEVER run `bun test`.	10	package script. Never use `bun test`.
12	- Run the narrowest useful verification for a change. For broad or merge-ready	11	- Use the narrowest meaningful verification while iterating. Merge-ready gates
13	work the full gates are `bun run format:check`, `lint`, `typecheck`, `test`.	12	are `bun run format:check`, `lint`, `typecheck`, and `test`.
14	- Run `bun run format` before opening a PR, staging only files your branch	13	- Run `bun run format` before a PR and include only files owned by the branch.
15	touched (leave unrelated files `format` rewrites out of the PR).	14	- User-visible work requires a specific e2e run, a browsable dev instance, and
		15	the recording or trace showing what to inspect. Add a scenario when none
		16	covers the changed behavior.
16		17
17	## Handing Back Work: Evidence, Not Assertions	18	## Service emulators
18		19
19	"Done" is something the user can open, not a claim. When work changes what a user	20	Tests and demos that need an upstream API, OAuth/OIDC provider, or webhook use
20	sees, hand back: an e2e run that proves it (link the specific run with one line	21	the published `@executor-js/emulate` emulators rather than stubs. They provide
21	on what to look at, not a wall of green), the dev server left running so they can	22	wire-level state, real-shaped credentials, OpenAPI descriptions, and request
22	poke at it, and the paths worth trying by hand including ones no scenario covers	23	ledgers. Create per-run hosted instances through the service's
23	yet. If no scenario covers the change, write one, and embed its recording in the	24	`/_emulate/instances` control route.
24	PR. The machinery runs in reverse too: seed an environment into a state
25	(reproduce a bug live, stage data) and hand across the link.
26		25
27	## Service Emulators	26	Emulate is a separate project. Make emulator changes there and consume the
		27	published package here; never re-vendor or create a parallel fake inside
		28	executor.
28		29
29	For any test or demo needing an upstream API, OAuth/OIDC provider, or webhook	30	## Engineering boundaries
30	source, use the `@executor-js/emulate` emulators instead of writing a stub:
31	wire-level and stateful, real SDKs run unmodified, each serves a full OpenAPI
32	spec, mints real-shaped credentials, and records every call in a request ledger
33	to assert against. Hosted on emulators.dev (create a per-run instance via
34	`POST https://<service>.emulators.dev/_emulate/instances`). See the `emulate`
35	skill (`.claude/skills/emulate/SKILL.md`). They are a standalone project
36	(`github.com/UsefulSoftwareCo/emulate`) consumed here as the published package:
37	full autonomy to change, publish, and deploy them on their `main`; don't re-vendor.
38
39	## Attribution
40
41	No AI/Claude/Anthropic/Co-Authored-By attribution in commits, messages, PRs, or
42	generated files. PR titles and descriptions go to a public repo: no internal info
43	or specific names unless explicitly stated.
44
45	## Collaboration Notes
46
47	- The user uses speech-to-text; infer likely intent from odd wording, ask only
48	when needed.
49	- Code is cheap to write: no time estimates, implementation time isn't a blocker.
50	- Never use em-dashes anywhere. Use commas, colons, parentheses, or separate
51	sentences.
52
53	## Reference Repos
54
55	Repos in `.reference` (Effect, effect-atom, …) are available for patterns. Clone
56	a given Git URL into `.reference` and pull latest before using it.
57
58	## Engineering Priorities
59		31
60	- Prefer correctness and predictable behavior over short-term convenience.	32	- Prefer correctness and predictable behavior over short-term convenience.
61	- Preserve runtime behavior when changing lint, typing, or test structure.	33	- Preserve runtime behavior during lint, typing, and test-structure changes.
62	- Keep package boundaries clear; use public package exports, not cross-package	34	- Use public package exports; never cross package boundaries with relative
63	relative imports.	35	imports.
64	- Extract shared logic only when the shared behavior is real; avoid broad generic	36	- Extract shared logic only for genuinely shared behavior. Avoid generic
65	abstractions for one-off duplication.	37	abstractions for one-off duplication.
		38	- Public PRs, commits, generated files, and documentation contain no private
		39	names, internal context, customer-derived data, or AI attribution.
66		40
67	## Package Roles	41	## Package ownership
68		42
69	- `packages/core/sdk`: core contracts, plugin wiring, scopes, integrations, secrets,	43	- `packages/core/sdk`: contracts, plugin wiring, scopes, integrations, secrets,
70	policies, fixtures. `@executor-js/sdk/http-auth` carries the shared auth-method	44	policies, and fixtures. Shared HTTP auth vocabulary lives in
71	vocabulary the HTTP protocol plugins compose (core never imports it, keeping it	45	`@executor-js/sdk/http-auth`; core remains carrier-agnostic.
72	carrier-agnostic).
73	- `packages/core/storage-*`: storage adapters and test support.	46	- `packages/core/storage-*`: storage adapters and test support.
74	- `packages/plugins/*`: protocol and provider plugins; their runtime, React, API,	47	- `packages/plugins/*`: protocol/provider implementations and their runtime,
75	and testing helpers live with the owning plugin.	48	React, API, and testing helpers.
76	- `packages/react`: shared React UI and atom/client integration.	49	- `packages/react`: shared React UI and client/atom integration.
77	- `packages/hosts/mcp`: MCP host surface.	50	- `packages/hosts/mcp`: MCP host surface.
78	- `packages/kernel/*`: execution runtimes and code execution substrate.	51	- `packages/kernel/*`: execution runtimes and code-execution substrate.
79	- `apps/{local,cloud,cli,desktop}`: product entry points composing the packages.	52	- `apps/{local,cloud,cli,desktop}`: product composition roots.
80
81	## Other
82		53
83	Note mistakes in MISTAKES.md, missing context or tools in DESIRES.md, and env	54	Record mistakes in `MISTAKES.md`, missing capabilities in `DESIRES.md`, and
84	learnings in LEARNINGS.md.	55	environment discoveries in `LEARNINGS.md`.

e2e/AGENTS.md+42−280

@@ -1,280 +1,42 @@
1	# Writing e2e scenarios	1	# E2E policy
2		2
3	A scenario is ONE user-meaningful product journey, written once against the	3	Detailed target setup, commands, ports, desktop VM operation, endpoint
4	`Target` interface and run on every deployment that supports its capabilities.	4	discovery, viewer usage, and recording workflows live in
5	Tests are black-box: drive the product only through public surfaces (typed	5	[RUNNING.md](RUNNING.md). Read only the relevant section for the task.
6	API, web UI, MCP, CLI). Never import app internals, never poke the DB, never	6
7	modify product code or stubs — if the product or stub blocks you, STOP and	7	## Scenario structure
8	report the blocker instead of working around it.	8
9		9	- Put cross-target scenarios in `scenarios/*.test.ts`; target-specific suites
10	The test source is the review artifact. A reviewer judges correctness by	10	stay under their target directory.
11	reading the test; write it so it reads as a spec. Assertions are plain vitest	11	- Declare capabilities by yielding services from `src/services.ts` (`Target`,
12	`expect` (use the message argument for intent). Browser runs additionally	12	`Api`, `Browser`, `Mcp`, `Billing`, `Cli`, `Telemetry`, and others). Do not
13	produce a Playwright trace, video, and step screenshots for debugging.	13	add a separate capability list. Unsupported targets skip and record why.
14		14	- Create fresh identities and organizations through `Target`. Never depend on
15	## File placement	15	shared state or customer-derived fixtures.
16		16	- Clean up created resources with `Effect.ensuring`; trailing cleanup statements
17	Scenario directories map to vitest projects (`vitest.config.ts` is the	17	are insufficient because mid-test failures must not leak state.
18	authoritative list of targets and what each one includes):	18	- Assert through the real typed API, browser, MCP, CLI, emulator ledger, or
19		19	exported telemetry surface. Do not reach into implementation internals to
20	- `scenarios/*.test.ts` — cross-target; runs on cloud + selfhost by default,	20	make a scenario pass.
21	and selected files also run on selfhost-docker and cloudflare	21
22	- `cloud/*.test.ts` — cloud-only (e.g. billing, WorkOS-session UI, telemetry)	22	## Browser and telemetry
23	- `selfhost/*.test.ts` — selfhost-only (also runs on selfhost-docker)	23
24	- `cloudflare/*.test.ts` — the Cloudflare self-host worker	24	- Wrap browser work in `browser.session(identity, ...)` and use `step` labels
25	- `local/*.test.ts` — the single-user local app; each scenario boots its own	25	phrased as user actions. Prefer role-based locators and wait for navigation or
26	`executor web`	26	network settling after actions that change pages.
27	- `cli/*.test.ts` — the supervised CLI daemon inside guest VMs	27	- Browser sessions produce screenshots, video, and a Playwright trace. Do not
28	- `desktop/`, `desktop-packaged/`, `desktop-vm/` — see Desktop targets below	28	add separate recording machinery for ordinary scenarios.
29		29	- Telemetry assertions target spans that reached the motel store, not values
30	## Anatomy	30	stamped only inside application memory. Export is eventually consistent, so
31		31	use the provided polling helpers.
32	```ts	32
33	import { expect } from "@effect/vitest";	33	## Isolation and quality
34	import { Effect } from "effect";	34
35	import { composePluginApi } from "@executor-js/api/server";	35	- Never hardcode ports or attach to an arbitrary existing server. Use the
36	import { scenario } from "../src/scenario";	36	checkout's claimed port block or explicit `E2E_<TARGET>_URL` attachment.
37	import { Api, Target } from "../src/services";	37	- Test the user-visible contract and important failure path, not selectors or
38		38	implementation details alone.
39	const coreApi = composePluginApi([] as const); // tools/integrations/connections/providers/executions/oauth/policies	39	- Run the narrowest relevant target while iterating. Before handoff, provide the
40		40	viewer/run URL, recording or trace, and the exact outcome to inspect.
41	scenario(	41	- Do not weaken, skip, or rewrite a failing assertion merely to make the suite
42	"Tools · a fresh workspace advertises the built-in tools",	42	green. Fix the product, fixture, or scenario contract.
43	{}, // options: { timeout?: number; skip?: string (reason — registers as skipped) }
44	Effect.gen(function* () {
45	const target = yield* Target;
46	const { client } = yield* Api;
47	const identity = yield* target.newIdentity(); // fresh isolated user+org
48	const api = yield* client(coreApi, identity); // typed HttpApiClient
49	const tools = yield* api.tools.list({ query: {} });
50	expect(tools.length, "at least one tool is exposed").toBeGreaterThan(0);
51	}),
52	);
53	```
54
55	- A scenario declares what it needs by yielding services from
56	`src/services.ts` (`Target`, `Api`, `Browser`, `Mcp`, `Billing`, `Cli`,
57	`Telemetry`, …). There is no `needs` list: yielding a service the current
58	target can't provide skips the test and records why in `skipped.json`.
59	- Which target provides what (from `targets/*.ts`): `api` — everything except
60	local and the desktop targets; `browser` — cloud, selfhost, selfhost-docker,
61	cloudflare, local; `mcp-oauth` — cloud, selfhost, selfhost-docker,
62	cloudflare (dev-auth on cloudflare, so no real consent hop); `billing` —
63	cloud only. `Telemetry` and `Autumn` appear when the suite booted motel /
64	the Autumn emulator (cloud).
65	- Resources created in a test must be cleaned up with `Effect.ensuring` (a
66	finalizer), not trailing statements — a mid-test failure must not leak state
67	into the shared instance.
68
69	## Browser scenarios
70
71	```ts
72	const target = yield * Target;
73	const browser = yield * Browser;
74	const identity = yield * target.newIdentity(); // logged in, has an org
75	// or newIdentity({ org: false }) for the onboarding flow
76	yield *
77	browser.session(identity, async ({ page, step }) => {
78	await step("A fresh user lands on the integrations page", async () => {
79	await page.goto("/", { waitUntil: "networkidle" });
80	await page.getByText("Integrations").first().waitFor();
81	});
82	});
83	```
84
85	- `step(label, fn)` names a Playwright trace group and saves a screenshot —
86	label steps as user actions ("Open the org switcher"), not selectors.
87	- The session records video (mp4) + a full Playwright trace into the run's
88	artifact dir; a failure saves `failure.png` automatically.
89	- Prefer role-based locators (`getByRole("menuitem", ...)`) — text locators
90	often match the look-alike trigger button in the bottom bar.
91	- After an action that navigates, wait for the URL/network to settle before
92	opening menus: `await page.waitForLoadState("networkidle")`.
93	- The stub user renders as "Test User" / `test@example.com`.
94
95	## MCP scenarios
96
97	```ts
98	const mcp = yield * Mcp;
99	const session = mcp.session(identity);
100	const tools = yield * session.listTools(); // OAuth happens headlessly here
101	const r = yield * session.call("execute", { code: "return 1 + 1;" });
102	// human-in-the-loop: session.approvePaused(r.text) resumes a paused execution
103	```
104
105	## Telemetry scenarios (cloud)
106
107	The suite boots a motel OTLP store and points the target's real exporter at
108	it, so a scenario can assert on the spans the server ACTUALLY exported —
109	the layer where "observability silently went dark" bugs live (an attribute
110	stamped on a span the exporter never carries looks identical to health).
111
112	```ts
113	const telemetry = yield * Telemetry; // skips when motel didn't boot
114	const span =
115	yield *
116	telemetry.expectSpan({
117	operation: "executor.tool.execute",
118	attributes: { "mcp.tool.name": failAddress }, // exact match, values stringified
119	});
120	expect(span.span.tags["executor.tool.outcome"]).toBe("fail");
121	```
122
123	- `expectSpan` polls (~20s): exporters batch, so arrival is
124	eventually-consistent — "the span reaches the store, soon" IS the contract.
125	- The cloud globalsetup boots motel automatically; `bun run motel` runs the
126	same store standalone (browse it, or point a dev server's exporter at it).
127	- Spec gotcha for fixtures: give operations explicit `tags` — tool addresses
128	are `group.leaf`, and an untagged op derives its group from the URL path,
129	so `/fail` does NOT produce a `.fail`-suffixed address.
130	- Prior art: `cloud/telemetry-contract.test.ts`.
131
132	## Running
133
134	```sh
135	cd e2e
136	bun run test # boots both dev servers, runs cloud + selfhost
137	bun run test:cloud # one target (also: test:selfhost, test:selfhost-docker,
138	# test:cloudflare, test:local, test:desktop, test:watch)
139	bun run ports # print THIS checkout's derived ports
140	bun run summary # pass/fail digest per target from runs/
141	# attach to an already-running server while iterating (use `bun run ports` URLs):
142	E2E_CLOUD_URL=http://127.0.0.1:<port> ../node_modules/.bin/vitest run --project cloud <file>
143	E2E_SELFHOST_URL=http://localhost:<port> ../node_modules/.bin/vitest run --project selfhost <file>
144	```
145
146	For interactive work against a live instance (boot, mint identities, typed API
147	calls, MCP calls, emulator ledger) use the dev CLI: `bun run cli` — full
148	command list in [RUNNING.md](../RUNNING.md).
149
150	Ports are claimed at boot (see `src/ports.ts`): each checkout hashes its repo
151	root to a preferred block, atomically locks it (a held lock port makes races
152	impossible), and walks to the next free block if it's locked or squatted — so
153	concurrent suites in different worktrees can never collide or attach to each
154	other's servers. `bun run ports` shows the preferred block; the boot log says
155	if a suite moved. `E2E_*_PORT` env vars pin ports explicitly (no probing) and
156	`E2E_<TARGET>_URL` attaches to a running instance.
157
158	Each run writes `runs/<target>/<slug>/result.json` plus any browser artifacts
159	(trace.zip / session.mp4 / screenshots). `bun run serve` hosts the scenario ×
160	target matrix; a run page links the trace into Playwright's trace viewer. The
161	viewer itself is a Vite/React SPA in `viewer/` (rebuilt into `runs/` by
162	`bun run viewer:build`); `bun e2e/scripts/pr-media.ts runs/<target>/<slug>`
163	turns a run's recording into PR-ready markdown.
164
165	When handing results to the user, follow the evidence contract in the root
166	[AGENTS.md](../AGENTS.md) (direct run links + a live instance + what to try);
167	[RUNNING.md](../RUNNING.md) has the current sharing/demo mechanics.
168
169	## Desktop targets (the app on real OSes, filmed)
170
171	The packaged desktop app runs as its own targets, each landing in its own
172	`runs/<target>/` bucket with a video. One shared scenario (`desktop-vm/`) and the
173	shared driver (`src/vm/desktop.ts`) + setup plumbing (`setup/desktop-vm.ts`); one
174	project + globalsetup per guest OS.
175
176	- `desktop-packaged` — the real electron-builder bundle on THIS machine's
177	display (the supervised-daemon attach path). Needs a logged-in GUI session.
178	- `desktop-macos` / `desktop-linux` — the same bundle inside a guest VM,
179	driven over CDP from the host and filmed. The globalsetup boots the guest
180	(tart), builds + pushes the bundle, brings the app up with
181	`--remote-debugging-port`, forwards it, and the scenario connects + drives +
182	records. Provisioned automatically — or attach to a running guest with
183	`E2E_DESKTOP_VM_IP=<ip>`:
184
185	```sh
186	vitest run --project desktop-macos # or desktop-linux
187	```
188
189	- `desktop-windows` — same scenario, but ATTACHES to a long-lived dockur
190	Windows guest over an SSH jump instead of provisioning one (no bundle build).
191
192	There are also `cli-macos` / `cli-linux` / `cli-windows` projects (the
193	supervised CLI daemon inside a guest VM, `cli/*.test.ts` +
194	`scenarios/restart-persistence.test.ts`): the globalsetup provisions the VM
195	and `executor service install`s the daemon; `restart()` reboots the guest for
196	real, proving the boot-time auto-start path. tart for macOS/Linux, EC2 for
197	Windows.
198
199	macOS-guest gotchas (VNC login first, single-instance lock vs a host
200	Executor.app, guest log paths): see [notes/testing-on-mac.md](notes/testing-on-mac.md).
201
202	The guests run tart `--no-graphics` (no host window, never steals focus) but
203	still have a usable display:
204
205	- macOS: the base image's autologin reaches a real Aqua session
206	(WindowServer/Dock/Finder). Launch the app INTO it with `sudo launchctl asuser
207	<uid> …` (a plain SSH spawn lands in a non-GUI session); the unsigned arm64
208	bundle is ad-hoc `codesign`'d in the guest; `screencapture` films it.
209	- linux: no window server, so the app renders into an `Xvfb` display with a
210	minimal WM (`openbox` — without it the electron window never maps); the window
211	maps tiny (10x10) so the globalsetup `xdotool`-resizes it to fill, and ffmpeg
212	`x11grab` films it. `--no-sandbox` (the chrome-sandbox needs setuid root).
213
214	Base images (`admin`/`admin`): `executor-macos-base` (cirruslabs sequoia, autologin)
215	and `executor-linux-base` (cirruslabs ubuntu + Xvfb/ffmpeg/openbox/xdotool +
216	electron runtime libs). The bundle's `executor` binary is cross-compiled for the
217	guest (`BUN_TARGET`), and electron-builder's `dir` target assembles the unpacked
218	app on macOS — so both bundles build on this Mac.
219
220	Note: `desktop-packaged`'s `guiAvailable()` probe (`launchctl managername`) reads
221	"Background" over SSH even when Aqua is up, so it's host-only; the VM targets gate
222	on a CDP page target instead.
223
224	## Discovering endpoints
225
226	- The full OpenAPI spec: `curl http://127.0.0.1:<cloud port>/api/openapi.json`
227	(cloud; port from `bun run ports`).
228	- The typed client mirrors it: `client.<group>.<endpoint>(...)` with groups
229	tools/integrations/connections/providers/executions/oauth/policies.
230	- To see payload shapes, read the API definitions under
231	`packages/core/api/src/<group>/api.ts` (READ ONLY — for shapes, not imports).
232
233	## Isolation rules
234
235	- Cloud: `newIdentity()` is a fresh user+org — you are isolated for free.
236	- Selfhost: everyone is the bootstrap admin. PREFIX every resource you create
237	with your scenario slug (e.g. policy pattern `policies-scn.*`) so parallel
238	scenarios don't collide, and don't assert on global counts (assert "contains
239	mine", not "length is 1").
240
241	## Quality bar
242
243	- The scenario name reads like a product guarantee ("Billing · the free plan
244	stops organization creation after 3"), not a test id.
245	- The test reads as a spec top-to-bottom; a reviewer should understand the
246	journey and the guarantee without running it.
247	- Assert outcomes the user cares about, not implementation details. No
248	tautologies (don't assert what the setup already guarantees). Assert on
249	values, not booleans — `expect(list).toContain(x)`, never
250	`expect(list.includes(x)).toBe(true)` — so failures show the data.
251	- Keep it deterministic: no sleeps; wait on conditions.
252
253	## Developer-session recordings (chat theater + desk)
254
255	Some scenarios are meant to be WATCHED — they show the product the way a
256	developer actually uses it. Three tiers, pick deliberately:
257
258	1. Chat theater (`src/clients/chat-theater.ts`): the default for
259	product-flow recordings. The "agent" is a chat renderer in a recorded
260	PTY; every tool spinner brackets a REAL mcporter MCP call (OAuth,
261	execute, approval resume). No inference, no third-party binary.
262	Exemplar: `scenarios/connect-handoff-session.test.ts`. Artifacts:
263	`terminal.cast` (the chat) + `session.mp4` (browser hops); the viewer
264	plays them in story order.
265	2. Replay brain + real client (`src/clients/replay-brain.ts`): when the
266	third-party CLIENT's behavior is under test (OpenCode/Claude Code
267	protocol handling). A scripted OpenAI-wire server plays the LLM; the
268	real client does everything else. Script by transcript inspection, never
269	turn counting.
270	3. Real-inference evals: a different axis (performance distributions,
271	not pass/fail). Not in this suite.
272
273	The Desk (`desk/`): films a scenario on one virtual Linux desktop — the
274	chat renderer in a visible xterm, the browser as a real headed window, one
275	ffmpeg x11grab. The film replaces session.mp4 in the run dir; the scenario
276	file is unchanged (chat-theater switches transports on `E2E_DESK=1`).
277
278	```
279	e2e/desk/run.sh [scenario] [project] # docker; first run builds + installs
280	```

e2e/RUNNING.md+283−0

@@ -0,0 +1,283 @@
	1	# E2E runbook
	2
	3	This is the detailed scenario-authoring and target-operation reference. The
	4	always-on invariants are in [AGENTS.md](AGENTS.md).
	5
	6	A scenario is ONE user-meaningful product journey, written once against the
	7	`Target` interface and run on every deployment that supports its capabilities.
	8	Tests are black-box: drive the product only through public surfaces (typed
	9	API, web UI, MCP, CLI). Never import app internals, never poke the DB, never
	10	modify product code or stubs — if the product or stub blocks you, STOP and
	11	report the blocker instead of working around it.
	12
	13	The test source is the review artifact. A reviewer judges correctness by
	14	reading the test; write it so it reads as a spec. Assertions are plain vitest
	15	`expect` (use the message argument for intent). Browser runs additionally
	16	produce a Playwright trace, video, and step screenshots for debugging.
	17
	18	## File placement
	19
	20	Scenario directories map to vitest projects (`vitest.config.ts` is the
	21	authoritative list of targets and what each one includes):
	22
	23	- `scenarios/*.test.ts` — cross-target; runs on cloud + selfhost by default,
	24	and selected files also run on selfhost-docker and cloudflare
	25	- `cloud/*.test.ts` — cloud-only (e.g. billing, WorkOS-session UI, telemetry)
	26	- `selfhost/*.test.ts` — selfhost-only (also runs on selfhost-docker)
	27	- `cloudflare/*.test.ts` — the Cloudflare self-host worker
	28	- `local/*.test.ts` — the single-user local app; each scenario boots its own
	29	`executor web`
	30	- `cli/*.test.ts` — the supervised CLI daemon inside guest VMs
	31	- `desktop/`, `desktop-packaged/`, `desktop-vm/` — see Desktop targets below
	32
	33	## Anatomy
	34
	35	```ts
	36	import { expect } from "@effect/vitest";
	37	import { Effect } from "effect";
	38	import { composePluginApi } from "@executor-js/api/server";
	39	import { scenario } from "../src/scenario";
	40	import { Api, Target } from "../src/services";
	41
	42	const coreApi = composePluginApi([] as const); // tools/integrations/connections/providers/executions/oauth/policies
	43
	44	scenario(
	45	"Tools · a fresh workspace advertises the built-in tools",
	46	{}, // options: { timeout?: number; skip?: string (reason — registers as skipped) }
	47	Effect.gen(function* () {
	48	const target = yield* Target;
	49	const { client } = yield* Api;
	50	const identity = yield* target.newIdentity(); // fresh isolated user+org
	51	const api = yield* client(coreApi, identity); // typed HttpApiClient
	52	const tools = yield* api.tools.list({ query: {} });
	53	expect(tools.length, "at least one tool is exposed").toBeGreaterThan(0);
	54	}),
	55	);
	56	```
	57
	58	- A scenario declares what it needs by yielding services from
	59	`src/services.ts` (`Target`, `Api`, `Browser`, `Mcp`, `Billing`, `Cli`,
	60	`Telemetry`, …). There is no `needs` list: yielding a service the current
	61	target can't provide skips the test and records why in `skipped.json`.
	62	- Which target provides what (from `targets/*.ts`): `api` — everything except
	63	local and the desktop targets; `browser` — cloud, selfhost, selfhost-docker,
	64	cloudflare, local; `mcp-oauth` — cloud, selfhost, selfhost-docker,
	65	cloudflare (dev-auth on cloudflare, so no real consent hop); `billing` —
	66	cloud only. `Telemetry` and `Autumn` appear when the suite booted motel /
	67	the Autumn emulator (cloud).
	68	- Resources created in a test must be cleaned up with `Effect.ensuring` (a
	69	finalizer), not trailing statements — a mid-test failure must not leak state
	70	into the shared instance.
	71
	72	## Browser scenarios
	73
	74	```ts
	75	const target = yield * Target;
	76	const browser = yield * Browser;
	77	const identity = yield * target.newIdentity(); // logged in, has an org
	78	// or newIdentity({ org: false }) for the onboarding flow
	79	yield *
	80	browser.session(identity, async ({ page, step }) => {
	81	await step("A fresh user lands on the integrations page", async () => {
	82	await page.goto("/", { waitUntil: "networkidle" });
	83	await page.getByText("Integrations").first().waitFor();
	84	});
	85	});
	86	```
	87
	88	- `step(label, fn)` names a Playwright trace group and saves a screenshot —
	89	label steps as user actions ("Open the org switcher"), not selectors.
	90	- The session records video (mp4) + a full Playwright trace into the run's
	91	artifact dir; a failure saves `failure.png` automatically.
	92	- Prefer role-based locators (`getByRole("menuitem", ...)`) — text locators
	93	often match the look-alike trigger button in the bottom bar.
	94	- After an action that navigates, wait for the URL/network to settle before
	95	opening menus: `await page.waitForLoadState("networkidle")`.
	96	- The stub user renders as "Test User" / `test@example.com`.
	97
	98	## MCP scenarios
	99
	100	```ts
	101	const mcp = yield * Mcp;
	102	const session = mcp.session(identity);
	103	const tools = yield * session.listTools(); // OAuth happens headlessly here
	104	const r = yield * session.call("execute", { code: "return 1 + 1;" });
	105	// human-in-the-loop: session.approvePaused(r.text) resumes a paused execution
	106	```
	107
	108	## Telemetry scenarios (cloud)
	109
	110	The suite boots a motel OTLP store and points the target's real exporter at
	111	it, so a scenario can assert on the spans the server ACTUALLY exported —
	112	the layer where "observability silently went dark" bugs live (an attribute
	113	stamped on a span the exporter never carries looks identical to health).
	114
	115	```ts
	116	const telemetry = yield * Telemetry; // skips when motel didn't boot
	117	const span =
	118	yield *
	119	telemetry.expectSpan({
	120	operation: "executor.tool.execute",
	121	attributes: { "mcp.tool.name": failAddress }, // exact match, values stringified
	122	});
	123	expect(span.span.tags["executor.tool.outcome"]).toBe("fail");
	124	```
	125
	126	- `expectSpan` polls (~20s): exporters batch, so arrival is
	127	eventually-consistent — "the span reaches the store, soon" IS the contract.
	128	- The cloud globalsetup boots motel automatically; `bun run motel` runs the
	129	same store standalone (browse it, or point a dev server's exporter at it).
	130	- Spec gotcha for fixtures: give operations explicit `tags` — tool addresses
	131	are `group.leaf`, and an untagged op derives its group from the URL path,
	132	so `/fail` does NOT produce a `.fail`-suffixed address.
	133	- Prior art: `cloud/telemetry-contract.test.ts`.
	134
	135	## Running
	136
	137	```sh
	138	cd e2e
	139	bun run test # boots both dev servers, runs cloud + selfhost
	140	bun run test:cloud # one target (also: test:selfhost, test:selfhost-docker,
	141	# test:cloudflare, test:local, test:desktop, test:watch)
	142	bun run ports # print THIS checkout's derived ports
	143	bun run summary # pass/fail digest per target from runs/
	144	# attach to an already-running server while iterating (use `bun run ports` URLs):
	145	E2E_CLOUD_URL=http://127.0.0.1:<port> ../node_modules/.bin/vitest run --project cloud <file>
	146	E2E_SELFHOST_URL=http://localhost:<port> ../node_modules/.bin/vitest run --project selfhost <file>
	147	```
	148
	149	For interactive work against a live instance (boot, mint identities, typed API
	150	calls, MCP calls, emulator ledger) use the dev CLI: `bun run cli` — full
	151	command list in [RUNNING.md](../RUNNING.md).
	152
	153	Ports are claimed at boot (see `src/ports.ts`): each checkout hashes its repo
	154	root to a preferred block, atomically locks it (a held lock port makes races
	155	impossible), and walks to the next free block if it's locked or squatted — so
	156	concurrent suites in different worktrees can never collide or attach to each
	157	other's servers. `bun run ports` shows the preferred block; the boot log says
	158	if a suite moved. `E2E_*_PORT` env vars pin ports explicitly (no probing) and
	159	`E2E_<TARGET>_URL` attaches to a running instance.
	160
	161	Each run writes `runs/<target>/<slug>/result.json` plus any browser artifacts
	162	(trace.zip / session.mp4 / screenshots). `bun run serve` hosts the scenario ×
	163	target matrix; a run page links the trace into Playwright's trace viewer. The
	164	viewer itself is a Vite/React SPA in `viewer/` (rebuilt into `runs/` by
	165	`bun run viewer:build`); `bun e2e/scripts/pr-media.ts runs/<target>/<slug>`
	166	turns a run's recording into PR-ready markdown.
	167
	168	When handing results to the user, follow the evidence contract in the root
	169	[AGENTS.md](../AGENTS.md) (direct run links + a live instance + what to try);
	170	[RUNNING.md](../RUNNING.md) has the current sharing/demo mechanics.
	171
	172	## Desktop targets (the app on real OSes, filmed)
	173
	174	The packaged desktop app runs as its own targets, each landing in its own
	175	`runs/<target>/` bucket with a video. One shared scenario (`desktop-vm/`) and the
	176	shared driver (`src/vm/desktop.ts`) + setup plumbing (`setup/desktop-vm.ts`); one
	177	project + globalsetup per guest OS.
	178
	179	- `desktop-packaged` — the real electron-builder bundle on THIS machine's
	180	display (the supervised-daemon attach path). Needs a logged-in GUI session.
	181	- `desktop-macos` / `desktop-linux` — the same bundle inside a guest VM,
	182	driven over CDP from the host and filmed. The globalsetup boots the guest
	183	(tart), builds + pushes the bundle, brings the app up with
	184	`--remote-debugging-port`, forwards it, and the scenario connects + drives +
	185	records. Provisioned automatically — or attach to a running guest with
	186	`E2E_DESKTOP_VM_IP=<ip>`:
	187
	188	```sh
	189	vitest run --project desktop-macos # or desktop-linux
	190	```
	191
	192	- `desktop-windows` — same scenario, but ATTACHES to a long-lived dockur
	193	Windows guest over an SSH jump instead of provisioning one (no bundle build).
	194
	195	There are also `cli-macos` / `cli-linux` / `cli-windows` projects (the
	196	supervised CLI daemon inside a guest VM, `cli/*.test.ts` +
	197	`scenarios/restart-persistence.test.ts`): the globalsetup provisions the VM
	198	and `executor service install`s the daemon; `restart()` reboots the guest for
	199	real, proving the boot-time auto-start path. tart for macOS/Linux, EC2 for
	200	Windows.
	201
	202	macOS-guest gotchas (VNC login first, single-instance lock vs a host
	203	Executor.app, guest log paths): see [notes/testing-on-mac.md](notes/testing-on-mac.md).
	204
	205	The guests run tart `--no-graphics` (no host window, never steals focus) but
	206	still have a usable display:
	207
	208	- macOS: the base image's autologin reaches a real Aqua session
	209	(WindowServer/Dock/Finder). Launch the app INTO it with `sudo launchctl asuser
	210	<uid> …` (a plain SSH spawn lands in a non-GUI session); the unsigned arm64
	211	bundle is ad-hoc `codesign`'d in the guest; `screencapture` films it.
	212	- linux: no window server, so the app renders into an `Xvfb` display with a
	213	minimal WM (`openbox` — without it the electron window never maps); the window
	214	maps tiny (10x10) so the globalsetup `xdotool`-resizes it to fill, and ffmpeg
	215	`x11grab` films it. `--no-sandbox` (the chrome-sandbox needs setuid root).
	216
	217	Base images (`admin`/`admin`): `executor-macos-base` (cirruslabs sequoia, autologin)
	218	and `executor-linux-base` (cirruslabs ubuntu + Xvfb/ffmpeg/openbox/xdotool +
	219	electron runtime libs). The bundle's `executor` binary is cross-compiled for the
	220	guest (`BUN_TARGET`), and electron-builder's `dir` target assembles the unpacked
	221	app on macOS — so both bundles build on this Mac.
	222
	223	Note: `desktop-packaged`'s `guiAvailable()` probe (`launchctl managername`) reads
	224	"Background" over SSH even when Aqua is up, so it's host-only; the VM targets gate
	225	on a CDP page target instead.
	226
	227	## Discovering endpoints
	228
	229	- The full OpenAPI spec: `curl http://127.0.0.1:<cloud port>/api/openapi.json`
	230	(cloud; port from `bun run ports`).
	231	- The typed client mirrors it: `client.<group>.<endpoint>(...)` with groups
	232	tools/integrations/connections/providers/executions/oauth/policies.
	233	- To see payload shapes, read the API definitions under
	234	`packages/core/api/src/<group>/api.ts` (READ ONLY — for shapes, not imports).
	235
	236	## Isolation rules
	237
	238	- Cloud: `newIdentity()` is a fresh user+org — you are isolated for free.
	239	- Selfhost: everyone is the bootstrap admin. PREFIX every resource you create
	240	with your scenario slug (e.g. policy pattern `policies-scn.*`) so parallel
	241	scenarios don't collide, and don't assert on global counts (assert "contains
	242	mine", not "length is 1").
	243
	244	## Quality bar
	245
	246	- The scenario name reads like a product guarantee ("Billing · the free plan
	247	stops organization creation after 3"), not a test id.
	248	- The test reads as a spec top-to-bottom; a reviewer should understand the
	249	journey and the guarantee without running it.
	250	- Assert outcomes the user cares about, not implementation details. No
	251	tautologies (don't assert what the setup already guarantees). Assert on
	252	values, not booleans — `expect(list).toContain(x)`, never
	253	`expect(list.includes(x)).toBe(true)` — so failures show the data.
	254	- Keep it deterministic: no sleeps; wait on conditions.
	255
	256	## Developer-session recordings (chat theater + desk)
	257
	258	Some scenarios are meant to be WATCHED — they show the product the way a
	259	developer actually uses it. Three tiers, pick deliberately:
	260
	261	1. Chat theater (`src/clients/chat-theater.ts`): the default for
	262	product-flow recordings. The "agent" is a chat renderer in a recorded
	263	PTY; every tool spinner brackets a REAL mcporter MCP call (OAuth,
	264	execute, approval resume). No inference, no third-party binary.
	265	Exemplar: `scenarios/connect-handoff-session.test.ts`. Artifacts:
	266	`terminal.cast` (the chat) + `session.mp4` (browser hops); the viewer
	267	plays them in story order.
	268	2. Replay brain + real client (`src/clients/replay-brain.ts`): when the
	269	third-party CLIENT's behavior is under test (OpenCode/Claude Code
	270	protocol handling). A scripted OpenAI-wire server plays the LLM; the
	271	real client does everything else. Script by transcript inspection, never
	272	turn counting.
	273	3. Real-inference evals: a different axis (performance distributions,
	274	not pass/fail). Not in this suite.
	275
	276	The Desk (`desk/`): films a scenario on one virtual Linux desktop — the
	277	chat renderer in a visible xterm, the browser as a real headed window, one
	278	ffmpeg x11grab. The film replaces session.mp4 in the run dir; the scenario
	279	file is unchanged (chat-theater switches transports on `E2E_DESK=1`).
	280
	281	```
	282	e2e/desk/run.sh [scenario] [project] # docker; first run builds + installs
	283	```