Trim agent instructions

Rhys Sullivan committed 3 months ago

Commit a3c0b0a1 parent 2226e723 files · +365 −349Browse files

Unified · Split

3 files

AGENTS.md +40 −69
e2e/AGENTS.md +42 −280
e2e/RUNNING.md +283 −0

@@ -1,84 +1,55 @@
1		−# AGENTS.md
	1	+# Executor policy
2	2
3		−Principles: the contracts that stay true while implementations churn. For how to
4		−run, boot, share, or navigate things today, see [RUNNING.md](RUNNING.md); for
5		−e2e scenarios, [e2e/AGENTS.md](e2e/AGENTS.md). Run `bun run bootstrap` first in
6		−any fresh checkout or worktree.
	3	+Stable contracts live here. Current setup and server mechanics are in
	4	+[RUNNING.md](RUNNING.md); e2e invariants are in
	5	+[e2e/AGENTS.md](e2e/AGENTS.md). Run `bun run bootstrap` in every fresh rift.
7	6
8		−## Task Completion
	7	+## Verification and evidence
9	8
10		−- Tests use Effect Vitest. Run scoped tests with `vitest run ...`. `bun run test`
11		− is fine (it delegates to Vitest); NEVER run `bun test`.
12		−- Run the narrowest useful verification for a change. For broad or merge-ready
13		− work the full gates are `bun run format:check`, `lint`, `typecheck`, `test`.
14		−- Run `bun run format` before opening a PR, staging only files your branch
15		− touched (leave unrelated files `format` rewrites out of the PR).
	9	+- Tests use Effect Vitest. Run scoped tests with `vitest run ...` or the
	10	+ package script. Never use `bun test`.
	11	+- Use the narrowest meaningful verification while iterating. Merge-ready gates
	12	+ are `bun run format:check`, `lint`, `typecheck`, and `test`.
	13	+- Run `bun run format` before a PR and include only files owned by the branch.
	14	+- User-visible work requires a specific e2e run, a browsable dev instance, and
	15	+ the recording or trace showing what to inspect. Add a scenario when none
	16	+ covers the changed behavior.
16	17
17		−## Handing Back Work: Evidence, Not Assertions
	18	+## Service emulators
18	19
19		−"Done" is something the user can open, not a claim. When work changes what a user
20		−sees, hand back: an e2e run that proves it (link the specific run with one line
21		−on what to look at, not a wall of green), the dev server left running so they can
22		−poke at it, and the paths worth trying by hand including ones no scenario covers
23		−yet. If no scenario covers the change, write one, and embed its recording in the
24		−PR. The machinery runs in reverse too: seed an environment into a state
25		−(reproduce a bug live, stage data) and hand across the link.
	20	+Tests and demos that need an upstream API, OAuth/OIDC provider, or webhook use
	21	+the published `@executor-js/emulate` emulators rather than stubs. They provide
	22	+wire-level state, real-shaped credentials, OpenAPI descriptions, and request
	23	+ledgers. Create per-run hosted instances through the service's
	24	+`/_emulate/instances` control route.
26	25
27		−## Service Emulators
	26	+Emulate is a separate project. Make emulator changes there and consume the
	27	+published package here; never re-vendor or create a parallel fake inside
	28	+executor.
28	29
29		−For any test or demo needing an upstream API, OAuth/OIDC provider, or webhook
30		−source, use the `@executor-js/emulate` emulators instead of writing a stub:
31		−wire-level and stateful, real SDKs run unmodified, each serves a full OpenAPI
32		−spec, mints real-shaped credentials, and records every call in a request ledger
33		−to assert against. Hosted on emulators.dev (create a per-run instance via
34		−`POST https://<service>.emulators.dev/_emulate/instances`). See the `emulate`
35		−skill (`.claude/skills/emulate/SKILL.md`). They are a standalone project
36		−(`github.com/UsefulSoftwareCo/emulate`) consumed here as the published package:
37		−full autonomy to change, publish, and deploy them on their `main`; don't re-vendor.
38		−
39		−## Attribution
40		−
41		−No AI/Claude/Anthropic/Co-Authored-By attribution in commits, messages, PRs, or
42		−generated files. PR titles and descriptions go to a public repo: no internal info
43		−or specific names unless explicitly stated.
44		−
45		−## Collaboration Notes
46		−
47		−- The user uses speech-to-text; infer likely intent from odd wording, ask only
48		− when needed.
49		−- Code is cheap to write: no time estimates, implementation time isn't a blocker.
50		−- Never use em-dashes anywhere. Use commas, colons, parentheses, or separate
51		− sentences.
52		−
53		−## Reference Repos
54		−
55		−Repos in `.reference` (Effect, effect-atom, …) are available for patterns. Clone
56		−a given Git URL into `.reference` and pull latest before using it.
57		−
58		−## Engineering Priorities
	30	+## Engineering boundaries
59	31
60	32	- Prefer correctness and predictable behavior over short-term convenience.
61		−- Preserve runtime behavior when changing lint, typing, or test structure.
62		−- Keep package boundaries clear; use public package exports, not cross-package
63		− relative imports.
64		−- Extract shared logic only when the shared behavior is real; avoid broad generic
	33	+- Preserve runtime behavior during lint, typing, and test-structure changes.
	34	+- Use public package exports; never cross package boundaries with relative
	35	+ imports.
	36	+- Extract shared logic only for genuinely shared behavior. Avoid generic
65	37	abstractions for one-off duplication.
	38	+- Public PRs, commits, generated files, and documentation contain no private
	39	+ names, internal context, customer-derived data, or AI attribution.
66	40
67		−## Package Roles
	41	+## Package ownership
68	42
69		−- `packages/core/sdk`: core contracts, plugin wiring, scopes, integrations, secrets,
70		− policies, fixtures. `@executor-js/sdk/http-auth` carries the shared auth-method
71		− vocabulary the HTTP protocol plugins compose (core never imports it, keeping it
72		− carrier-agnostic).
	43	+- `packages/core/sdk`: contracts, plugin wiring, scopes, integrations, secrets,
	44	+ policies, and fixtures. Shared HTTP auth vocabulary lives in
	45	+ `@executor-js/sdk/http-auth`; core remains carrier-agnostic.
73	46	- `packages/core/storage-*`: storage adapters and test support.
74		−- `packages/plugins/*`: protocol and provider plugins; their runtime, React, API,
75		− and testing helpers live with the owning plugin.
76		−- `packages/react`: shared React UI and atom/client integration.
	47	+- `packages/plugins/*`: protocol/provider implementations and their runtime,
	48	+ React, API, and testing helpers.
	49	+- `packages/react`: shared React UI and client/atom integration.
77	50	- `packages/hosts/mcp`: MCP host surface.
78		−- `packages/kernel/*`: execution runtimes and code execution substrate.
79		−- `apps/{local,cloud,cli,desktop}`: product entry points composing the packages.
80		−
81		−## Other
	51	+- `packages/kernel/*`: execution runtimes and code-execution substrate.
	52	+- `apps/{local,cloud,cli,desktop}`: product composition roots.
82	53
83		−Note mistakes in MISTAKES.md, missing context or tools in DESIRES.md, and env
84		−learnings in LEARNINGS.md.
	54	+Record mistakes in `MISTAKES.md`, missing capabilities in `DESIRES.md`, and
	55	+environment discoveries in `LEARNINGS.md`.

e2e/AGENTS.md+42−280

@@ -1,280 +1,42 @@
1		−# Writing e2e scenarios
2		−
3		−A scenario is ONE user-meaningful product journey, written once against the
4		−`Target` interface and run on every deployment that supports its capabilities.
5		−Tests are black-box: drive the product only through public surfaces (typed
6		−API, web UI, MCP, CLI). Never import app internals, never poke the DB, never
7		−modify product code or stubs — if the product or stub blocks you, STOP and
8		−report the blocker instead of working around it.
9		−
10		−The test source is the review artifact. A reviewer judges correctness by
11		−reading the test; write it so it reads as a spec. Assertions are plain vitest
12		−`expect` (use the message argument for intent). Browser runs additionally
13		−produce a Playwright trace, video, and step screenshots for debugging.
14		−
15		−## File placement
16		−
17		−Scenario directories map to vitest projects (`vitest.config.ts` is the
18		−authoritative list of targets and what each one includes):
19		−
20		−- `scenarios/*.test.ts` — cross-target; runs on cloud + selfhost by default,
21		− and selected files also run on selfhost-docker and cloudflare
22		−- `cloud/*.test.ts` — cloud-only (e.g. billing, WorkOS-session UI, telemetry)
23		−- `selfhost/*.test.ts` — selfhost-only (also runs on selfhost-docker)
24		−- `cloudflare/*.test.ts` — the Cloudflare self-host worker
25		−- `local/*.test.ts` — the single-user local app; each scenario boots its own
26		− `executor web`
27		−- `cli/*.test.ts` — the supervised CLI daemon inside guest VMs
28		−- `desktop/`, `desktop-packaged/`, `desktop-vm/` — see Desktop targets below
29		−
30		−## Anatomy
31		−
32		−```ts
33		−import { expect } from "@effect/vitest";
34		−import { Effect } from "effect";
35		−import { composePluginApi } from "@executor-js/api/server";
36		−import { scenario } from "../src/scenario";
37		−import { Api, Target } from "../src/services";
38		−
39		−const coreApi = composePluginApi([] as const); // tools/integrations/connections/providers/executions/oauth/policies
40		−
41		−scenario(
42		− "Tools · a fresh workspace advertises the built-in tools",
43		− {}, // options: { timeout?: number; skip?: string (reason — registers as skipped) }
44		− Effect.gen(function* () {
45		− const target = yield* Target;
46		− const { client } = yield* Api;
47		− const identity = yield* target.newIdentity(); // fresh isolated user+org
48		− const api = yield* client(coreApi, identity); // typed HttpApiClient
49		− const tools = yield* api.tools.list({ query: {} });
50		− expect(tools.length, "at least one tool is exposed").toBeGreaterThan(0);
51		− }),
52		−);
53		−```
54		−
55		−- A scenario declares what it needs by yielding services from
56		− `src/services.ts` (`Target`, `Api`, `Browser`, `Mcp`, `Billing`, `Cli`,
57		− `Telemetry`, …). There is no `needs` list: yielding a service the current
58		− target can't provide skips the test and records why in `skipped.json`.
59		−- Which target provides what (from `targets/*.ts`): `api` — everything except
60		− local and the desktop targets; `browser` — cloud, selfhost, selfhost-docker,
61		− cloudflare, local; `mcp-oauth` — cloud, selfhost, selfhost-docker,
62		− cloudflare (dev-auth on cloudflare, so no real consent hop); `billing` —
63		− cloud only. `Telemetry` and `Autumn` appear when the suite booted motel /
64		− the Autumn emulator (cloud).
65		−- Resources created in a test must be cleaned up with `Effect.ensuring` (a
66		− finalizer), not trailing statements — a mid-test failure must not leak state
67		− into the shared instance.
68		−
69		−## Browser scenarios
70		−
71		−```ts
72		−const target = yield * Target;
73		−const browser = yield * Browser;
74		−const identity = yield * target.newIdentity(); // logged in, has an org
75		−// or newIdentity({ org: false }) for the onboarding flow
76		−yield *
77		− browser.session(identity, async ({ page, step }) => {
78		− await step("A fresh user lands on the integrations page", async () => {
79		− await page.goto("/", { waitUntil: "networkidle" });
80		− await page.getByText("Integrations").first().waitFor();
81		− });
82		− });
83		−```
84		−
85		−- `step(label, fn)` names a Playwright trace group and saves a screenshot —
86		− label steps as user actions ("Open the org switcher"), not selectors.
87		−- The session records video (mp4) + a full Playwright trace into the run's
88		− artifact dir; a failure saves `failure.png` automatically.
89		−- Prefer role-based locators (`getByRole("menuitem", ...)`) — text locators
90		− often match the look-alike trigger button in the bottom bar.
91		−- After an action that navigates, wait for the URL/network to settle before
92		− opening menus: `await page.waitForLoadState("networkidle")`.
93		−- The stub user renders as "Test User" / `test@example.com`.
94		−
95		−## MCP scenarios
96		−
97		−```ts
98		−const mcp = yield * Mcp;
99		−const session = mcp.session(identity);
100		−const tools = yield * session.listTools(); // OAuth happens headlessly here
101		−const r = yield * session.call("execute", { code: "return 1 + 1;" });
102		−// human-in-the-loop: session.approvePaused(r.text) resumes a paused execution
103		−```
104		−
105		−## Telemetry scenarios (cloud)
106		−
107		−The suite boots a motel OTLP store and points the target's real exporter at
108		−it, so a scenario can assert on the spans the server ACTUALLY exported —
109		−the layer where "observability silently went dark" bugs live (an attribute
110		−stamped on a span the exporter never carries looks identical to health).
111		−
112		−```ts
113		−const telemetry = yield * Telemetry; // skips when motel didn't boot
114		−const span =
115		− yield *
116		− telemetry.expectSpan({
117		− operation: "executor.tool.execute",
118		− attributes: { "mcp.tool.name": failAddress }, // exact match, values stringified
119		− });
120		−expect(span.span.tags["executor.tool.outcome"]).toBe("fail");
121		−```
122		−
123		−- `expectSpan` polls (~20s): exporters batch, so arrival is
124		− eventually-consistent — "the span reaches the store, soon" IS the contract.
125		−- The cloud globalsetup boots motel automatically; `bun run motel` runs the
126		− same store standalone (browse it, or point a dev server's exporter at it).
127		−- Spec gotcha for fixtures: give operations explicit `tags` — tool addresses
128		− are `group.leaf`, and an untagged op derives its group from the URL path,
129		− so `/fail` does NOT produce a `.fail`-suffixed address.
130		−- Prior art: `cloud/telemetry-contract.test.ts`.
131		−
132		−## Running
133		−
134		−```sh
135		−cd e2e
136		−bun run test # boots both dev servers, runs cloud + selfhost
137		−bun run test:cloud # one target (also: test:selfhost, test:selfhost-docker,
138		− # test:cloudflare, test:local, test:desktop, test:watch)
139		−bun run ports # print THIS checkout's derived ports
140		−bun run summary # pass/fail digest per target from runs/
141		−# attach to an already-running server while iterating (use `bun run ports` URLs):
142		−E2E_CLOUD_URL=http://127.0.0.1:<port> ../node_modules/.bin/vitest run --project cloud <file>
143		−E2E_SELFHOST_URL=http://localhost:<port> ../node_modules/.bin/vitest run --project selfhost <file>
144		−```
145		−
146		−For interactive work against a live instance (boot, mint identities, typed API
147		−calls, MCP calls, emulator ledger) use the dev CLI: `bun run cli` — full
148		−command list in [RUNNING.md](../RUNNING.md).
149		−
150		−Ports are claimed at boot (see `src/ports.ts`): each checkout hashes its repo
151		−root to a preferred block, atomically locks it (a held lock port makes races
152		−impossible), and walks to the next free block if it's locked or squatted — so
153		−concurrent suites in different worktrees can never collide or attach to each
154		−other's servers. `bun run ports` shows the preferred block; the boot log says
155		−if a suite moved. `E2E_*_PORT` env vars pin ports explicitly (no probing) and
156		−`E2E_<TARGET>_URL` attaches to a running instance.
157		−
158		−Each run writes `runs/<target>/<slug>/result.json` plus any browser artifacts
159		−(trace.zip / session.mp4 / screenshots). `bun run serve` hosts the scenario ×
160		−target matrix; a run page links the trace into Playwright's trace viewer. The
161		−viewer itself is a Vite/React SPA in `viewer/` (rebuilt into `runs/` by
162		−`bun run viewer:build`); `bun e2e/scripts/pr-media.ts runs/<target>/<slug>`
163		−turns a run's recording into PR-ready markdown.
164		−
165		−When handing results to the user, follow the evidence contract in the root
166		−[AGENTS.md](../AGENTS.md) (direct run links + a live instance + what to try);
167		−[RUNNING.md](../RUNNING.md) has the current sharing/demo mechanics.
168		−
169		−## Desktop targets (the app on real OSes, filmed)
170		−
171		−The packaged desktop app runs as its own targets, each landing in its own
172		−`runs/<target>/` bucket with a video. One shared scenario (`desktop-vm/`) and the
173		−shared driver (`src/vm/desktop.ts`) + setup plumbing (`setup/desktop-vm.ts`); one
174		−project + globalsetup per guest OS.
175		−
176		−- `desktop-packaged` — the real electron-builder bundle on THIS machine's
177		− display (the supervised-daemon attach path). Needs a logged-in GUI session.
178		−- `desktop-macos` / `desktop-linux` — the same bundle inside a guest VM,
179		− driven over CDP from the host and filmed. The globalsetup boots the guest
180		− (tart), builds + pushes the bundle, brings the app up with
181		− `--remote-debugging-port`, forwards it, and the scenario connects + drives +
182		− records. Provisioned automatically — or attach to a running guest with
183		− `E2E_DESKTOP_VM_IP=<ip>`:
184		−
185		− ```sh
186		− vitest run --project desktop-macos # or desktop-linux
187		− ```
188		−
189		−- `desktop-windows` — same scenario, but ATTACHES to a long-lived dockur
190		− Windows guest over an SSH jump instead of provisioning one (no bundle build).
191		−
192		−There are also `cli-macos` / `cli-linux` / `cli-windows` projects (the
193		−supervised CLI daemon inside a guest VM, `cli/*.test.ts` +
194		−`scenarios/restart-persistence.test.ts`): the globalsetup provisions the VM
195		−and `executor service install`s the daemon; `restart()` reboots the guest for
196		−real, proving the boot-time auto-start path. tart for macOS/Linux, EC2 for
197		−Windows.
198		−
199		−macOS-guest gotchas (VNC login first, single-instance lock vs a host
200		−Executor.app, guest log paths): see [notes/testing-on-mac.md](notes/testing-on-mac.md).
201		−
202		−The guests run tart `--no-graphics` (no host window, never steals focus) but
203		−still have a usable display:
204		−
205		−- macOS: the base image's autologin reaches a real Aqua session
206		− (WindowServer/Dock/Finder). Launch the app INTO it with `sudo launchctl asuser
207		−<uid> …` (a plain SSH spawn lands in a non-GUI session); the unsigned arm64
208		− bundle is ad-hoc `codesign`'d in the guest; `screencapture` films it.
209		−- linux: no window server, so the app renders into an `Xvfb` display with a
210		− minimal WM (`openbox` — without it the electron window never maps); the window
211		− maps tiny (10x10) so the globalsetup `xdotool`-resizes it to fill, and ffmpeg
212		− `x11grab` films it. `--no-sandbox` (the chrome-sandbox needs setuid root).
213		−
214		−Base images (`admin`/`admin`): `executor-macos-base` (cirruslabs sequoia, autologin)
215		−and `executor-linux-base` (cirruslabs ubuntu + Xvfb/ffmpeg/openbox/xdotool +
216		−electron runtime libs). The bundle's `executor` binary is cross-compiled for the
217		−guest (`BUN_TARGET`), and electron-builder's `dir` target assembles the unpacked
218		−app on macOS — so both bundles build on this Mac.
219		−
220		−Note: `desktop-packaged`'s `guiAvailable()` probe (`launchctl managername`) reads
221		−"Background" over SSH even when Aqua is up, so it's host-only; the VM targets gate
222		−on a CDP page target instead.
223		−
224		−## Discovering endpoints
225		−
226		−- The full OpenAPI spec: `curl http://127.0.0.1:<cloud port>/api/openapi.json`
227		− (cloud; port from `bun run ports`).
228		−- The typed client mirrors it: `client.<group>.<endpoint>(...)` with groups
229		− tools/integrations/connections/providers/executions/oauth/policies.
230		−- To see payload shapes, read the API definitions under
231		− `packages/core/api/src/<group>/api.ts` (READ ONLY — for shapes, not imports).
232		−
233		−## Isolation rules
234		−
235		−- Cloud: `newIdentity()` is a fresh user+org — you are isolated for free.
236		−- Selfhost: everyone is the bootstrap admin. PREFIX every resource you create
237		− with your scenario slug (e.g. policy pattern `policies-scn.*`) so parallel
238		− scenarios don't collide, and don't assert on global counts (assert "contains
239		− mine", not "length is 1").
240		−
241		−## Quality bar
242		−
243		−- The scenario name reads like a product guarantee ("Billing · the free plan
244		− stops organization creation after 3"), not a test id.
245		−- The test reads as a spec top-to-bottom; a reviewer should understand the
246		− journey and the guarantee without running it.
247		−- Assert outcomes the user cares about, not implementation details. No
248		− tautologies (don't assert what the setup already guarantees). Assert on
249		− values, not booleans — `expect(list).toContain(x)`, never
250		− `expect(list.includes(x)).toBe(true)` — so failures show the data.
251		−- Keep it deterministic: no sleeps; wait on conditions.
252		−
253		−## Developer-session recordings (chat theater + desk)
254		−
255		−Some scenarios are meant to be WATCHED — they show the product the way a
256		−developer actually uses it. Three tiers, pick deliberately:
257		−
258		−1. Chat theater (`src/clients/chat-theater.ts`): the default for
259		− product-flow recordings. The "agent" is a chat renderer in a recorded
260		− PTY; every tool spinner brackets a REAL mcporter MCP call (OAuth,
261		− execute, approval resume). No inference, no third-party binary.
262		− Exemplar: `scenarios/connect-handoff-session.test.ts`. Artifacts:
263		− `terminal.cast` (the chat) + `session.mp4` (browser hops); the viewer
264		− plays them in story order.
265		−2. Replay brain + real client (`src/clients/replay-brain.ts`): when the
266		− third-party CLIENT's behavior is under test (OpenCode/Claude Code
267		− protocol handling). A scripted OpenAI-wire server plays the LLM; the
268		− real client does everything else. Script by transcript inspection, never
269		− turn counting.
270		−3. Real-inference evals: a different axis (performance distributions,
271		− not pass/fail). Not in this suite.
272		−
273		−The Desk (`desk/`): films a scenario on one virtual Linux desktop — the
274		−chat renderer in a visible xterm, the browser as a real headed window, one
275		−ffmpeg x11grab. The film replaces session.mp4 in the run dir; the scenario
276		−file is unchanged (chat-theater switches transports on `E2E_DESK=1`).
277		−
278		−```
279		−e2e/desk/run.sh [scenario] [project] # docker; first run builds + installs
280		−```
	1	+# E2E policy
	2	+
	3	+Detailed target setup, commands, ports, desktop VM operation, endpoint
	4	+discovery, viewer usage, and recording workflows live in
	5	+[RUNNING.md](RUNNING.md). Read only the relevant section for the task.
	6	+
	7	+## Scenario structure
	8	+
	9	+- Put cross-target scenarios in `scenarios/*.test.ts`; target-specific suites
	10	+ stay under their target directory.
	11	+- Declare capabilities by yielding services from `src/services.ts` (`Target`,
	12	+ `Api`, `Browser`, `Mcp`, `Billing`, `Cli`, `Telemetry`, and others). Do not
	13	+ add a separate capability list. Unsupported targets skip and record why.
	14	+- Create fresh identities and organizations through `Target`. Never depend on
	15	+ shared state or customer-derived fixtures.
	16	+- Clean up created resources with `Effect.ensuring`; trailing cleanup statements
	17	+ are insufficient because mid-test failures must not leak state.
	18	+- Assert through the real typed API, browser, MCP, CLI, emulator ledger, or
	19	+ exported telemetry surface. Do not reach into implementation internals to
	20	+ make a scenario pass.
	21	+
	22	+## Browser and telemetry
	23	+
	24	+- Wrap browser work in `browser.session(identity, ...)` and use `step` labels
	25	+ phrased as user actions. Prefer role-based locators and wait for navigation or
	26	+ network settling after actions that change pages.
	27	+- Browser sessions produce screenshots, video, and a Playwright trace. Do not
	28	+ add separate recording machinery for ordinary scenarios.
	29	+- Telemetry assertions target spans that reached the motel store, not values
	30	+ stamped only inside application memory. Export is eventually consistent, so
	31	+ use the provided polling helpers.
	32	+
	33	+## Isolation and quality
	34	+
	35	+- Never hardcode ports or attach to an arbitrary existing server. Use the
	36	+ checkout's claimed port block or explicit `E2E_<TARGET>_URL` attachment.
	37	+- Test the user-visible contract and important failure path, not selectors or
	38	+ implementation details alone.
	39	+- Run the narrowest relevant target while iterating. Before handoff, provide the
	40	+ viewer/run URL, recording or trace, and the exact outcome to inspect.
	41	+- Do not weaken, skip, or rewrite a failing assertion merely to make the suite
	42	+ green. Fix the product, fixture, or scenario contract.

e2e/RUNNING.md+283−0

@@ -0,0 +1,283 @@
	1	+# E2E runbook
	2	+
	3	+This is the detailed scenario-authoring and target-operation reference. The
	4	+always-on invariants are in [AGENTS.md](AGENTS.md).
	5	+
	6	+A scenario is ONE user-meaningful product journey, written once against the
	7	+`Target` interface and run on every deployment that supports its capabilities.
	8	+Tests are black-box: drive the product only through public surfaces (typed
	9	+API, web UI, MCP, CLI). Never import app internals, never poke the DB, never
	10	+modify product code or stubs — if the product or stub blocks you, STOP and
	11	+report the blocker instead of working around it.
	12	+
	13	+The test source is the review artifact. A reviewer judges correctness by
	14	+reading the test; write it so it reads as a spec. Assertions are plain vitest
	15	+`expect` (use the message argument for intent). Browser runs additionally
	16	+produce a Playwright trace, video, and step screenshots for debugging.
	17	+
	18	+## File placement
	19	+
	20	+Scenario directories map to vitest projects (`vitest.config.ts` is the
	21	+authoritative list of targets and what each one includes):
	22	+
	23	+- `scenarios/*.test.ts` — cross-target; runs on cloud + selfhost by default,
	24	+ and selected files also run on selfhost-docker and cloudflare
	25	+- `cloud/*.test.ts` — cloud-only (e.g. billing, WorkOS-session UI, telemetry)
	26	+- `selfhost/*.test.ts` — selfhost-only (also runs on selfhost-docker)
	27	+- `cloudflare/*.test.ts` — the Cloudflare self-host worker
	28	+- `local/*.test.ts` — the single-user local app; each scenario boots its own
	29	+ `executor web`
	30	+- `cli/*.test.ts` — the supervised CLI daemon inside guest VMs
	31	+- `desktop/`, `desktop-packaged/`, `desktop-vm/` — see Desktop targets below
	32	+
	33	+## Anatomy
	34	+
	35	+```ts
	36	+import { expect } from "@effect/vitest";
	37	+import { Effect } from "effect";
	38	+import { composePluginApi } from "@executor-js/api/server";
	39	+import { scenario } from "../src/scenario";
	40	+import { Api, Target } from "../src/services";
	41	+
	42	+const coreApi = composePluginApi([] as const); // tools/integrations/connections/providers/executions/oauth/policies
	43	+
	44	+scenario(
	45	+ "Tools · a fresh workspace advertises the built-in tools",
	46	+ {}, // options: { timeout?: number; skip?: string (reason — registers as skipped) }
	47	+ Effect.gen(function* () {
	48	+ const target = yield* Target;
	49	+ const { client } = yield* Api;
	50	+ const identity = yield* target.newIdentity(); // fresh isolated user+org
	51	+ const api = yield* client(coreApi, identity); // typed HttpApiClient
	52	+ const tools = yield* api.tools.list({ query: {} });
	53	+ expect(tools.length, "at least one tool is exposed").toBeGreaterThan(0);
	54	+ }),
	55	+);
	56	+```
	57	+
	58	+- A scenario declares what it needs by yielding services from
	59	+ `src/services.ts` (`Target`, `Api`, `Browser`, `Mcp`, `Billing`, `Cli`,
	60	+ `Telemetry`, …). There is no `needs` list: yielding a service the current
	61	+ target can't provide skips the test and records why in `skipped.json`.
	62	+- Which target provides what (from `targets/*.ts`): `api` — everything except
	63	+ local and the desktop targets; `browser` — cloud, selfhost, selfhost-docker,
	64	+ cloudflare, local; `mcp-oauth` — cloud, selfhost, selfhost-docker,
	65	+ cloudflare (dev-auth on cloudflare, so no real consent hop); `billing` —
	66	+ cloud only. `Telemetry` and `Autumn` appear when the suite booted motel /
	67	+ the Autumn emulator (cloud).
	68	+- Resources created in a test must be cleaned up with `Effect.ensuring` (a
	69	+ finalizer), not trailing statements — a mid-test failure must not leak state
	70	+ into the shared instance.
	71	+
	72	+## Browser scenarios
	73	+
	74	+```ts
	75	+const target = yield * Target;
	76	+const browser = yield * Browser;
	77	+const identity = yield * target.newIdentity(); // logged in, has an org
	78	+// or newIdentity({ org: false }) for the onboarding flow
	79	+yield *
	80	+ browser.session(identity, async ({ page, step }) => {
	81	+ await step("A fresh user lands on the integrations page", async () => {
	82	+ await page.goto("/", { waitUntil: "networkidle" });
	83	+ await page.getByText("Integrations").first().waitFor();
	84	+ });
	85	+ });
	86	+```
	87	+
	88	+- `step(label, fn)` names a Playwright trace group and saves a screenshot —
	89	+ label steps as user actions ("Open the org switcher"), not selectors.
	90	+- The session records video (mp4) + a full Playwright trace into the run's
	91	+ artifact dir; a failure saves `failure.png` automatically.
	92	+- Prefer role-based locators (`getByRole("menuitem", ...)`) — text locators
	93	+ often match the look-alike trigger button in the bottom bar.
	94	+- After an action that navigates, wait for the URL/network to settle before
	95	+ opening menus: `await page.waitForLoadState("networkidle")`.
	96	+- The stub user renders as "Test User" / `test@example.com`.
	97	+
	98	+## MCP scenarios
	99	+
	100	+```ts
	101	+const mcp = yield * Mcp;
	102	+const session = mcp.session(identity);
	103	+const tools = yield * session.listTools(); // OAuth happens headlessly here
	104	+const r = yield * session.call("execute", { code: "return 1 + 1;" });
	105	+// human-in-the-loop: session.approvePaused(r.text) resumes a paused execution
	106	+```
	107	+
	108	+## Telemetry scenarios (cloud)
	109	+
	110	+The suite boots a motel OTLP store and points the target's real exporter at
	111	+it, so a scenario can assert on the spans the server ACTUALLY exported —
	112	+the layer where "observability silently went dark" bugs live (an attribute
	113	+stamped on a span the exporter never carries looks identical to health).
	114	+
	115	+```ts
	116	+const telemetry = yield * Telemetry; // skips when motel didn't boot
	117	+const span =
	118	+ yield *
	119	+ telemetry.expectSpan({
	120	+ operation: "executor.tool.execute",
	121	+ attributes: { "mcp.tool.name": failAddress }, // exact match, values stringified
	122	+ });
	123	+expect(span.span.tags["executor.tool.outcome"]).toBe("fail");
	124	+```
	125	+
	126	+- `expectSpan` polls (~20s): exporters batch, so arrival is
	127	+ eventually-consistent — "the span reaches the store, soon" IS the contract.
	128	+- The cloud globalsetup boots motel automatically; `bun run motel` runs the
	129	+ same store standalone (browse it, or point a dev server's exporter at it).
	130	+- Spec gotcha for fixtures: give operations explicit `tags` — tool addresses
	131	+ are `group.leaf`, and an untagged op derives its group from the URL path,
	132	+ so `/fail` does NOT produce a `.fail`-suffixed address.
	133	+- Prior art: `cloud/telemetry-contract.test.ts`.
	134	+
	135	+## Running
	136	+
	137	+```sh
	138	+cd e2e
	139	+bun run test # boots both dev servers, runs cloud + selfhost
	140	+bun run test:cloud # one target (also: test:selfhost, test:selfhost-docker,
	141	+ # test:cloudflare, test:local, test:desktop, test:watch)
	142	+bun run ports # print THIS checkout's derived ports
	143	+bun run summary # pass/fail digest per target from runs/
	144	+# attach to an already-running server while iterating (use `bun run ports` URLs):
	145	+E2E_CLOUD_URL=http://127.0.0.1:<port> ../node_modules/.bin/vitest run --project cloud <file>
	146	+E2E_SELFHOST_URL=http://localhost:<port> ../node_modules/.bin/vitest run --project selfhost <file>
	147	+```
	148	+
	149	+For interactive work against a live instance (boot, mint identities, typed API
	150	+calls, MCP calls, emulator ledger) use the dev CLI: `bun run cli` — full
	151	+command list in [RUNNING.md](../RUNNING.md).
	152	+
	153	+Ports are claimed at boot (see `src/ports.ts`): each checkout hashes its repo
	154	+root to a preferred block, atomically locks it (a held lock port makes races
	155	+impossible), and walks to the next free block if it's locked or squatted — so
	156	+concurrent suites in different worktrees can never collide or attach to each
	157	+other's servers. `bun run ports` shows the preferred block; the boot log says
	158	+if a suite moved. `E2E_*_PORT` env vars pin ports explicitly (no probing) and
	159	+`E2E_<TARGET>_URL` attaches to a running instance.
	160	+
	161	+Each run writes `runs/<target>/<slug>/result.json` plus any browser artifacts
	162	+(trace.zip / session.mp4 / screenshots). `bun run serve` hosts the scenario ×
	163	+target matrix; a run page links the trace into Playwright's trace viewer. The
	164	+viewer itself is a Vite/React SPA in `viewer/` (rebuilt into `runs/` by
	165	+`bun run viewer:build`); `bun e2e/scripts/pr-media.ts runs/<target>/<slug>`
	166	+turns a run's recording into PR-ready markdown.
	167	+
	168	+When handing results to the user, follow the evidence contract in the root
	169	+[AGENTS.md](../AGENTS.md) (direct run links + a live instance + what to try);
	170	+[RUNNING.md](../RUNNING.md) has the current sharing/demo mechanics.
	171	+
	172	+## Desktop targets (the app on real OSes, filmed)
	173	+
	174	+The packaged desktop app runs as its own targets, each landing in its own
	175	+`runs/<target>/` bucket with a video. One shared scenario (`desktop-vm/`) and the
	176	+shared driver (`src/vm/desktop.ts`) + setup plumbing (`setup/desktop-vm.ts`); one
	177	+project + globalsetup per guest OS.
	178	+
	179	+- `desktop-packaged` — the real electron-builder bundle on THIS machine's
	180	+ display (the supervised-daemon attach path). Needs a logged-in GUI session.
	181	+- `desktop-macos` / `desktop-linux` — the same bundle inside a guest VM,
	182	+ driven over CDP from the host and filmed. The globalsetup boots the guest
	183	+ (tart), builds + pushes the bundle, brings the app up with
	184	+ `--remote-debugging-port`, forwards it, and the scenario connects + drives +
	185	+ records. Provisioned automatically — or attach to a running guest with
	186	+ `E2E_DESKTOP_VM_IP=<ip>`:
	187	+
	188	+ ```sh
	189	+ vitest run --project desktop-macos # or desktop-linux
	190	+ ```
	191	+
	192	+- `desktop-windows` — same scenario, but ATTACHES to a long-lived dockur
	193	+ Windows guest over an SSH jump instead of provisioning one (no bundle build).
	194	+
	195	+There are also `cli-macos` / `cli-linux` / `cli-windows` projects (the
	196	+supervised CLI daemon inside a guest VM, `cli/*.test.ts` +
	197	+`scenarios/restart-persistence.test.ts`): the globalsetup provisions the VM
	198	+and `executor service install`s the daemon; `restart()` reboots the guest for
	199	+real, proving the boot-time auto-start path. tart for macOS/Linux, EC2 for
	200	+Windows.
	201	+
	202	+macOS-guest gotchas (VNC login first, single-instance lock vs a host
	203	+Executor.app, guest log paths): see [notes/testing-on-mac.md](notes/testing-on-mac.md).
	204	+
	205	+The guests run tart `--no-graphics` (no host window, never steals focus) but
	206	+still have a usable display:
	207	+
	208	+- macOS: the base image's autologin reaches a real Aqua session
	209	+ (WindowServer/Dock/Finder). Launch the app INTO it with `sudo launchctl asuser
	210	+<uid> …` (a plain SSH spawn lands in a non-GUI session); the unsigned arm64
	211	+ bundle is ad-hoc `codesign`'d in the guest; `screencapture` films it.
	212	+- linux: no window server, so the app renders into an `Xvfb` display with a
	213	+ minimal WM (`openbox` — without it the electron window never maps); the window
	214	+ maps tiny (10x10) so the globalsetup `xdotool`-resizes it to fill, and ffmpeg
	215	+ `x11grab` films it. `--no-sandbox` (the chrome-sandbox needs setuid root).
	216	+
	217	+Base images (`admin`/`admin`): `executor-macos-base` (cirruslabs sequoia, autologin)
	218	+and `executor-linux-base` (cirruslabs ubuntu + Xvfb/ffmpeg/openbox/xdotool +
	219	+electron runtime libs). The bundle's `executor` binary is cross-compiled for the
	220	+guest (`BUN_TARGET`), and electron-builder's `dir` target assembles the unpacked
	221	+app on macOS — so both bundles build on this Mac.
	222	+
	223	+Note: `desktop-packaged`'s `guiAvailable()` probe (`launchctl managername`) reads
	224	+"Background" over SSH even when Aqua is up, so it's host-only; the VM targets gate
	225	+on a CDP page target instead.
	226	+
	227	+## Discovering endpoints
	228	+
	229	+- The full OpenAPI spec: `curl http://127.0.0.1:<cloud port>/api/openapi.json`
	230	+ (cloud; port from `bun run ports`).
	231	+- The typed client mirrors it: `client.<group>.<endpoint>(...)` with groups
	232	+ tools/integrations/connections/providers/executions/oauth/policies.
	233	+- To see payload shapes, read the API definitions under
	234	+ `packages/core/api/src/<group>/api.ts` (READ ONLY — for shapes, not imports).
	235	+
	236	+## Isolation rules
	237	+
	238	+- Cloud: `newIdentity()` is a fresh user+org — you are isolated for free.
	239	+- Selfhost: everyone is the bootstrap admin. PREFIX every resource you create
	240	+ with your scenario slug (e.g. policy pattern `policies-scn.*`) so parallel
	241	+ scenarios don't collide, and don't assert on global counts (assert "contains
	242	+ mine", not "length is 1").
	243	+
	244	+## Quality bar
	245	+
	246	+- The scenario name reads like a product guarantee ("Billing · the free plan
	247	+ stops organization creation after 3"), not a test id.
	248	+- The test reads as a spec top-to-bottom; a reviewer should understand the
	249	+ journey and the guarantee without running it.
	250	+- Assert outcomes the user cares about, not implementation details. No
	251	+ tautologies (don't assert what the setup already guarantees). Assert on
	252	+ values, not booleans — `expect(list).toContain(x)`, never
	253	+ `expect(list.includes(x)).toBe(true)` — so failures show the data.
	254	+- Keep it deterministic: no sleeps; wait on conditions.
	255	+
	256	+## Developer-session recordings (chat theater + desk)
	257	+
	258	+Some scenarios are meant to be WATCHED — they show the product the way a
	259	+developer actually uses it. Three tiers, pick deliberately:
	260	+
	261	+1. Chat theater (`src/clients/chat-theater.ts`): the default for
	262	+ product-flow recordings. The "agent" is a chat renderer in a recorded
	263	+ PTY; every tool spinner brackets a REAL mcporter MCP call (OAuth,
	264	+ execute, approval resume). No inference, no third-party binary.
	265	+ Exemplar: `scenarios/connect-handoff-session.test.ts`. Artifacts:
	266	+ `terminal.cast` (the chat) + `session.mp4` (browser hops); the viewer
	267	+ plays them in story order.
	268	+2. Replay brain + real client (`src/clients/replay-brain.ts`): when the
	269	+ third-party CLIENT's behavior is under test (OpenCode/Claude Code
	270	+ protocol handling). A scripted OpenAI-wire server plays the LLM; the
	271	+ real client does everything else. Script by transcript inspection, never
	272	+ turn counting.
	273	+3. Real-inference evals: a different axis (performance distributions,
	274	+ not pass/fail). Not in this suite.
	275	+
	276	+The Desk (`desk/`): films a scenario on one virtual Linux desktop — the
	277	+chat renderer in a visible xterm, the browser as a real headed window, one
	278	+ffmpeg x11grab. The film replaces session.mp4 in the run dir; the scenario
	279	+file is unchanged (chat-theater switches transports on `E2E_DESK=1`).
	280	+
	281	+```
	282	+e2e/desk/run.sh [scenario] [project] # docker; first run builds + installs
	283	+```