History

Drop the throwaway deploy-lab worker from the branch

Rhys Sullivan committed 4 months ago

Commit f1be2b91 parent dda3dad5 files · +18 −347Browse files

Unified · Split

5 files
  1. apps/cloud/DEPLOY.md +16 −33
  2. apps/cloud/scripts/deploy-lab/README.md +0 −86
  3. apps/cloud/scripts/deploy-lab/worker.ts +0 −192
  4. apps/cloud/scripts/deploy-lab/wrangler.jsonc +0 −34
  5. apps/cloud/scripts/promote-cloud.ts +2 −2
apps/cloud/DEPLOY.md+16−33
@@ -7,9 +7,10 @@ streaming sessions: a cutover mid-call drops in-flight MCP tool results. Staging · 6 unmodified lines
7the promotion bounds that blast radius and keeps a mere merge from disrupting7the promotion bounds that blast radius and keeps a mere merge from disrupting
8anything.8anything.
9 9
10The mechanics below were validated empirically on a throwaway lab worker10The mechanics below were validated empirically on a throwaway staging worker
11(`executor-cloud-deploy-lab`, see `scripts/deploy-lab/`); the "what happens"11that mounted the real session DO and patched transport with stub auth; the
12claims are measured, not assumed. The findings are summarized at the end.12"what happens" claims are measured, not assumed. The findings are summarized
13at the end.
13 14
14## The flow15## The flow
15 16
@@ -66,7 +67,7 @@ The script promotes exactly the version you name (`latest` resolves to the · 50 unmodified lines
66highest version number) and holds the remainder of a partial promotion on the67highest version number) and holds the remainder of a partial promotion on the
67current-live version. It refuses to run unless the resolved wrangler config68current-live version. It refuses to run unless the resolved wrangler config
68targets the `executor-cloud` worker (set `PROMOTE_CLOUD_ALLOW_NAME` to override69targets the `executor-cloud` worker (set `PROMOTE_CLOUD_ALLOW_NAME` to override
69for a non-production worker such as the deploy lab).70for a non-production worker).
70 71
71## Session version-affinity (zone ruleset, MUST be configured manually)72## Session version-affinity (zone ruleset, MUST be configured manually)
72 73
@@ -104,7 +105,7 @@ session to pin yet. Once the client echoes `mcp-session-id`, every subsequent · 31 unmodified lines
104request for that session is version-pinned to the worker version its DO landed105request for that session is version-pinned to the worker version its DO landed
105on.106on.
106 107
107Validated on the lab worker by sending `Cloudflare-Workers-Version-Key`108Validated on the staging worker by sending `Cloudflare-Workers-Version-Key`
108directly (12 requests per key across a 4-key set during a two-version gradual109directly (12 requests per key across a 4-key set during a two-version gradual
109deployment): each distinct key deterministically mapped to exactly one version110deployment): each distinct key deterministically mapped to exactly one version
110across all its requests, while unkeyed requests split across versions per the111across all its requests, while unkeyed requests split across versions per the
@@ -150,7 +151,7 @@ Constraints: · 39 unmodified lines
150 151
151## What the staged flow does and does not fix152## What the staged flow does and does not fix
152 153
153Measured on the lab worker:154Measured on the staging worker:
154 155
155- **Fixed, merges no longer disrupt anything.** `wrangler versions upload` does156- **Fixed, merges no longer disrupt anything.** `wrangler versions upload` does
156 not restart DOs; a session opened before an upload runs straight through it.157 not restart DOs; a session opened before an upload runs straight through it.
@@ -180,31 +181,13 @@ Measured on the lab worker: · 23 unmodified lines
180 and even a promotion loses only the single in-flight call, not the session or181 and even a promotion loses only the single in-flight call, not the session or
181 its history.182 its history.
182 183
183## The lab worker (`scripts/deploy-lab/`)184## Re-validating the mechanics
184 185
185`scripts/deploy-lab/` is a self-contained, re-runnable staging worker that186The measurements above came from a throwaway staging worker (a cut-down worker
186mounts the **real** `McpAgentSessionDOBase` + the patched `agents` transport187mounting the real `McpAgentSessionDOBase` + the patched `agents` transport with
187with stub auth and a trivial `execute` tool, so the versions / gradual-188stub auth and a trivial `execute` tool, deployed to a `workers.dev` subdomain,
188deployment / affinity / rollback mechanics can be validated against the real189never routed). To re-validate after a Cloudflare behavior change, rebuild the
189transport with no production secrets. Deploy it as190same shape: exercise affinity with `Cloudflare-Workers-Version-Key` across a
190`executor-cloud-deploy-lab` (workers_dev only, never routed):191two-version gradual deployment, fire an upload and a promotion mid-call, and
191 192roll back mid-session. Keep any such worker on `workers.dev` only and delete it
192```bash193when done.
193# from the repo root (bypasses apps/cloud's dist config redirect)
194wrangler deploy -c apps/cloud/scripts/deploy-lab/wrangler.jsonc --var LAB_VERSION:v1
195```
196
197It exposes `/mcp` (real streamable-HTTP MCP) and `/__lab/version` (echoes the
198serving worker version, for observing traffic splits and affinity). Delete it
199when done (`wrangler delete -c apps/cloud/scripts/deploy-lab/wrangler.jsonc`).
200
201> **workers.dev caveat (lab only).** The lab is reached over its
202> `*.workers.dev` subdomain, which had to be explicitly enabled once (Workers
203> dashboard -> the worker -> Settings -> Domains & Routes -> enable the
204> `workers.dev` route) before requests resolved; the config's `workers_dev:
205true` alone was not sufficient on first deploy. This does **not** apply to
206> production executor-cloud, which is served through configured **routes** on
207> `executor.sh`, not a workers.dev subdomain.
208
209See `scripts/deploy-lab/README.md` for the full experiment runbook and the
210measured results.
apps/cloud/scripts/deploy-lab/README.md+0−86
@@ -1,86 +0,0 @@
1# executor-cloud-deploy-lab
2
3A throwaway staging worker for validating the executor-cloud
4**upload -> staged-promotion** deploy mechanics against real Cloudflare
5deployment machinery (versions, gradual deployments, version affinity, rollback)
6and the patched `agents` worker<->DO MCP bridge. It mounts the **real** shared
7MCP session Durable Object base (`McpAgentSessionDOBase`) with the **real**
8patched transport (`patches/agents@0.17.3.patch`), and stubs only the heavy
9cloud seams (Postgres/Hyperdrive, WorkOS, Autumn, Sentry/OTEL) so it boots with
10no production secrets. See `worker.ts` for the stubs.
11
12**Never point this at production.** `workers_dev` only, no routes, no custom
13domains, no crons. It is safe to `wrangler delete` at any time.
14
15The narrative conclusions from these experiments live in
16[`../../DEPLOY.md`](../../DEPLOY.md); this file is the runbook that produced them.
17
18## Deploy / redeploy
19
20Deploy from the repo root so wrangler picks up this config directly (bypassing
21`apps/cloud`'s dist redirect). Each version is uploaded with a distinct
22`LAB_VERSION` var so every response echoes which worker version served it:
23
24```bash
25# Upload a version WITHOUT routing traffic (returns a version id):
26wrangler versions upload -c apps/cloud/scripts/deploy-lab/wrangler.jsonc --var LAB_VERSION:v6
27wrangler versions upload -c apps/cloud/scripts/deploy-lab/wrangler.jsonc --var LAB_VERSION:v7
28
29# Promote / split traffic:
30wrangler versions deploy -c apps/cloud/scripts/deploy-lab/wrangler.jsonc <v6-id>@90% <v7-id>@10% --yes
31```
32
33> **workers.dev must be enabled once.** `workers_dev: true` in the config was
34> not enough on first deploy: the `*.workers.dev` route had to be turned on
35> manually (Workers dashboard -> the worker -> Settings -> Domains & Routes ->
36> enable workers.dev) before requests resolved. Production is served through
37> `executor.sh` routes, so this caveat is lab-only.
38
39## Endpoints
40
41- `POST/GET/DELETE /mcp`: the real streamable-HTTP MCP surface (session create,
42 in-flight `tools/call`, POST-stream disconnect + GET replay). The single tool
43 `execute` sleeps `delayMs` then echoes `{ marker, servedByVersion }`, enough
44 to hold a ~30-60s call open across a mid-call deploy.
45- `GET /__lab/version`: echoes `{ servedByVersion, receivedVersionKey }` and an
46 `x-lab-version` header, for observing which worker version served a request
47 and whether an affinity key was received.
48
49## Experiment runbook and measured results
50
51All results below were measured against real Cloudflare deployments of this
52worker (2026-07-07).
53
54- **Exp3a, no affinity key.** 30 fresh connections over a 10/90 two-version
55 split: 28 hit v6, 2 hit v7. Confirms per-request (weighted) routing, a
56 stateless worker with no key does not stick to one version.
57- **Exp3b, with `Cloudflare-Workers-Version-Key`.** 12 requests per key across a
58 4-key set (alpha/bravo/charlie/delta) during the same split: each key mapped
59 to exactly one version for all 12 of its requests (alpha/bravo/delta -> v6,
60 charlie -> v7), and keys distributed across both versions. **Perfect per-key
61 stickiness**, this is the affinity mechanism DEPLOY.md's zone rule relies on.
62- **Exp3c, DO pinning.** One MCP session's DO answered on v6 for every request
63 including while the stateless front-worker hops varied version. DOs are
64 natively version-pinned.
65- **Exp4, rollback.** `versions deploy old@100` took effect within seconds
66 (10/10 subsequent requests on v6). A session that predated the rollback still
67 worked (`tools/list` -> 200) and its DO storage was intact.
68- **Exp5, upload mid-call.** `versions upload` fired during a ~30s in-flight
69 call caused **zero disruption**, the call completed and its marker was
70 delivered (`EXP5_UPLOAD_SAFE`). Uploads are free.
71- **Exp5b, promotion mid-call.** `versions deploy new@100` fired during a ~30s
72 in-flight call **killed the in-flight execution** (client saw the stream end
73 with no result, expected until durable-execution work lands). **But** the
74 session survived, the persisted event store survived the version swap, and a
75 reconnect GET replayed the prior events across the version boundary.
76
77**Conclusion.** The upload/promote split plus session affinity works as
78documented. Promotion is the only disruptive moment, and even then it loses only
79the single in-flight call, not the session or its replayable history. The replay
80machinery is version-swap-safe.
81
82## Cleanup
83
84```bash
85wrangler delete -c apps/cloud/scripts/deploy-lab/wrangler.jsonc
86```
apps/cloud/scripts/deploy-lab/worker.ts+0−192
@@ -1,192 +0,0 @@
1// ---------------------------------------------------------------------------
2// executor-cloud-deploy-lab, a CUT-DOWN staging worker used ONLY to validate
3// the upload->staged-promotion deploy mechanics (versions, gradual deployments,
4// version affinity, rollback) and the patched MCP bridge transport under real
5// Cloudflare deployment machinery.
6//
7// It mounts the REAL shared MCP session Durable Object base
8// (`McpAgentSessionDOBase` from @executor-js/cloudflare) with the REAL patched
9// `agents` transport (patches/agents@0.17.3.patch, the worker<->DO websocket
10// bridge). The heavy cloud seams (Postgres/Hyperdrive, WorkOS, Autumn billing,
11// Sentry/OTEL) are replaced with trivial stubs so the worker boots with NO
12// production secrets. The MCP transport path, session create, in-flight
13// tools/call, POST-stream disconnect + GET replay, runs for real.
14//
15// NEVER point this at production. workers_dev only, no routes/domains/crons.
16// ---------------------------------------------------------------------------
17import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
18import { Effect } from "effect";
19import * as z from "zod/v4";
20
21import {
22 McpAgentSessionDOBase,
23 type BuiltMcpServer,
24 type McpSessionInit,
25 type McpSessionProps,
26 type SessionDbHandle,
27 type SessionMeta,
28} from "@executor-js/cloudflare/mcp/agent-durable-object";
29import {
30 withVerifiedIdentityHeaders,
31 readElicitationMode,
32} from "@executor-js/cloudflare/mcp/do-headers";
33import { defaultMcpResource } from "@executor-js/host-mcp";
34
35// The version label is injected per-version via a `vars` override at
36// `wrangler versions upload` time (`--var LAB_VERSION:vN`). Every response
37// echoes it so a client can observe WHICH worker version served each request
38// (the crux of the affinity experiment) without needing dashboard access.
39interface LabEnv extends Cloudflare.Env {
40 readonly MCP_SESSION: DurableObjectNamespace;
41 readonly LAB_VERSION?: string;
42}
43
44// A no-op ExecutionEngine stub. The base only touches the engine on the
45// pause/resume approval path; a plain tools/call runs the tool handler
46// directly, so these are never exercised by the deploy experiments. Shape
47// mirrors packages/core/execution/src/engine.ts::ExecutionEngine.
48const makeStubEngine = () =>
49 // oxlint-disable-next-line executor/no-double-cast -- lab-only test double: the engine is never exercised by the deploy experiments (no pause/resume), so a structural stub standing in for the full ExecutionEngine is intentional.
50 ({
51 execute: () => Effect.succeed({ result: "lab-execute" }),
52 executeWithPause: () =>
53 Effect.succeed({ status: "completed", result: { result: "lab-execute" } }),
54 resume: () => Effect.succeed({ status: "completed", result: { result: "lab-resume" } }),
55 getPausedExecution: () => Effect.succeed(null),
56 pausedExecutionCount: () => Effect.succeed(0),
57 hasPausedExecutions: () => Effect.succeed(false),
58 getDescription: Effect.succeed("lab stub engine"),
59 }) as unknown as BuiltMcpServer["engine"];
60
61// The lab MCP server exposes a single `execute` tool that sleeps for a
62// caller-supplied duration and echoes a marker, enough to hold a ~60s
63// in-flight tool call open across a mid-call deploy (experiments 1 & 5).
64const buildLabMcpServer = (version: string): McpServer => {
65 const server = new McpServer({ name: "executor-cloud-deploy-lab", version: "0.0.1" });
66 server.registerTool(
67 "execute",
68 {
69 description: "Sleep for delayMs then echo marker. Lab-only, no real execution engine.",
70 inputSchema: {
71 code: z
72 .string()
73 .optional()
74 .describe("ignored (compat with the real execute tool signature)"),
75 delayMs: z.number().optional().describe("how long to sleep before responding"),
76 marker: z.string().optional().describe("string echoed back in the result"),
77 },
78 },
79 async (args: { code?: string; delayMs?: number; marker?: string }) => {
80 const delayMs = typeof args.delayMs === "number" ? args.delayMs : 0;
81 const marker = args.marker ?? "LAB_RESULT";
82 if (delayMs > 0) await new Promise((r) => setTimeout(r, delayMs));
83 return {
84 content: [
85 { type: "text" as const, text: JSON.stringify({ marker, servedByVersion: version }) },
86 ],
87 };
88 },
89 );
90 return server;
91};
92
93// The lab DO: the REAL base + REAL patched transport, trivial seams.
94export class McpSessionDOSqlite extends McpAgentSessionDOBase<LabEnv> {
95 protected openSessionDb(): SessionDbHandle {
96 return { end: () => undefined };
97 }
98
99 protected resolveSessionMeta(token: McpSessionInit): Effect.Effect<SessionMeta> {
100 return Effect.succeed({
101 organizationId: token.organizationId,
102 organizationName: "Deploy Lab Org",
103 userId: token.userId,
104 elicitationMode:
105 token.elicitationMode === "browser" || token.elicitationMode === "native"
106 ? token.elicitationMode
107 : "model",
108 resource: token.resource,
109 webOrigin: token.webOrigin,
110 });
111 }
112
113 protected buildMcpServer(): Effect.Effect<BuiltMcpServer> {
114 const version = (this.env as LabEnv).LAB_VERSION ?? "unknown";
115 return Effect.succeed({
116 mcpServer: buildLabMcpServer(version),
117 engine: makeStubEngine(),
118 });
119 }
120}
121
122// A stub auth layer: accepts ANY bearer, mints a fixed principal. This is the
123// ONLY thing standing in for the real WorkOS auth provider, the session
124// engine, DO, transport, and bridge below are the real, patched code.
125const LAB_PRINCIPAL = { accountId: "lab-user", organizationId: "lab-org" } as const;
126
127const serve = McpAgentSessionDOBase.serve("/mcp", {
128 binding: "MCP_SESSION",
129 transport: "streamable-http",
130}) as { fetch: (req: Request, env: LabEnv, ctx: ExecutionContext) => Promise<Response> };
131
132export default {
133 async fetch(request: Request, env: LabEnv, ctx: ExecutionContext): Promise<Response> {
134 const url = new URL(request.url);
135 const version = env.LAB_VERSION ?? "unknown";
136
137 // Version-echo endpoint: which worker version is serving THIS request.
138 // Used to observe stateless-worker version affinity during a gradual
139 // deployment. Also echoes the version-affinity header if the client set it.
140 if (url.pathname === "/__lab/version") {
141 return new Response(
142 JSON.stringify({
143 servedByVersion: version,
144 receivedVersionKey: request.headers.get("Cloudflare-Workers-Version-Key"),
145 }),
146 { headers: { "content-type": "application/json", "x-lab-version": version } },
147 );
148 }
149
150 if (url.pathname !== "/mcp") {
151 return new Response("lab worker: only /mcp and /__lab/version", { status: 404 });
152 }
153
154 if (request.method === "OPTIONS") {
155 return new Response(null, {
156 status: 204,
157 headers: {
158 "access-control-allow-origin": "*",
159 "access-control-allow-methods": "GET, POST, DELETE, OPTIONS",
160 "access-control-allow-headers":
161 "content-type, authorization, mcp-session-id, accept, mcp-protocol-version, last-event-id",
162 "access-control-expose-headers": "mcp-session-id",
163 },
164 });
165 }
166
167 // Stub auth: any bearer -> fixed principal. Build props exactly like the
168 // real agent-handler so the DO's onStart receives a valid McpSessionProps.
169 const props: McpSessionProps = {
170 session: {
171 organizationId: LAB_PRINCIPAL.organizationId,
172 userId: LAB_PRINCIPAL.accountId,
173 elicitationMode: readElicitationMode(request),
174 resource: defaultMcpResource,
175 webOrigin: url.origin,
176 },
177 };
178 (ctx as ExecutionContext & { props?: McpSessionProps }).props = props;
179
180 const forwarded = withVerifiedIdentityHeaders(request, LAB_PRINCIPAL, defaultMcpResource);
181 const response = await serve.fetch(forwarded, env, ctx);
182 // Stamp the serving worker version on every MCP response header too, so the
183 // client can log per-request affinity even for streamed responses.
184 const headers = new Headers(response.headers);
185 headers.set("x-lab-version", version);
186 return new Response(response.body, {
187 status: response.status,
188 statusText: response.statusText,
189 headers,
190 });
191 },
192};
apps/cloud/scripts/deploy-lab/wrangler.jsonc+0−34
@@ -1,34 +0,0 @@
1{
2 // STAGING deploy-mechanics lab worker. NOT production. workers_dev only —
3 // no routes, no custom_domain, no crons, no hyperdrive/r2/service bindings.
4 // See scripts/deploy-lab/README.md for the experiment runbook.
5 "$schema": "../../node_modules/wrangler/config-schema.json",
6 "name": "executor-cloud-deploy-lab",
7 "compatibility_date": "2025-04-01",
8 "compatibility_flags": ["nodejs_compat"],
9 "main": "worker.ts",
10 "workers_dev": true,
11 "observability": {
12 "enabled": true,
13 },
14 "durable_objects": {
15 "bindings": [
16 {
17 "name": "MCP_SESSION",
18 "class_name": "McpSessionDOSqlite",
19 },
20 ],
21 },
22 "migrations": [
23 {
24 "tag": "v1",
25 "new_sqlite_classes": ["McpSessionDOSqlite"],
26 },
27 ],
28 "placement": {
29 "region": "aws:us-east-1",
30 },
31 "vars": {
32 "LAB_VERSION": "unset",
33 },
34}
apps/cloud/scripts/promote-cloud.ts+2−2
@@ -39,12 +39,12 @@ import { spawnSync } from "node:child_process"; · 38 unmodified lines
39import { readFileSync } from "node:fs";39import { readFileSync } from "node:fs";
40 40
41// The generated dist config the build produced. Overridable via41// The generated dist config the build produced. Overridable via
42// PROMOTE_CLOUD_CONFIG for local/lab testing against a non-prod worker.42// PROMOTE_CLOUD_CONFIG for testing against a non-prod worker.
43const CONFIG = process.env.PROMOTE_CLOUD_CONFIG ?? "dist/server/wrangler.json";43const CONFIG = process.env.PROMOTE_CLOUD_CONFIG ?? "dist/server/wrangler.json";
44 44
45// Target-worker guard: refuse to run against anything but executor-cloud unless45// Target-worker guard: refuse to run against anything but executor-cloud unless
46// the operator explicitly names the worker they expect via46// the operator explicitly names the worker they expect via
47// PROMOTE_CLOUD_ALLOW_NAME (for the deploy lab / a non-prod worker). A promotion47// PROMOTE_CLOUD_ALLOW_NAME (for a non-prod worker). A promotion
48// routes live traffic and restarts DOs, so a misconfigured config that points at48// routes live traffic and restarts DOs, so a misconfigured config that points at
49// the wrong worker must fail loudly, not promote the wrong thing. `--yes` does49// the wrong worker must fail loudly, not promote the wrong thing. `--yes` does
50// NOT bypass this: it only skips wrangler's own interactive confirmation.50// NOT bypass this: it only skips wrangler's own interactive confirmation.

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t