Promote a named cloud version, decoupled from upload/migrate

A promote-only workflow_dispatch previously re-ran migrate and upload-cloud (building a fresh, un-canaried version) and then promoted whatever was newest, so advancing a canary could ship code that landed on main after the canary started. Decouple promotion: the promote-cloud job no longer needs upload, and migrate/upload/deploy-marketing run on push only. The version to promote is now an explicit promote_version input (a version id, or the literal latest as an opt-in), and promote-cloud.ts promotes exactly that version. Also guard the script against running with a wrangler config whose worker name is not executor-cloud.

Rhys Sullivan committed 4 months ago

Commit 5d11ea51 parent 4b260753 files · +182 −58Browse files

Unified · Split

3 files

.github/workflows/deploy.yml +49 −14
apps/cloud/DEPLOY.md +42 −16
apps/cloud/scripts/promote-cloud.ts +91 −28

.github/workflows/deploy.yml+49−14

 # blast radius of a promotion (the DO restart it causes) is staged and
 # operator-controlled.
 #
-# `promote_percent` (workflow_dispatch input): when set, the promote-cloud job
+# Promotion is DECOUPLED from upload: a `workflow_dispatch` that sets
-# runs and promotes the LATEST uploaded version to that percentage. Leave empty
+# `promote_percent` runs ONLY the promote-cloud job (no migrate, no upload, so
-# (the default, e.g. on a push) to upload only.
+# it never builds a new version). It promotes the version named by
+# `promote_version` (an explicit version id), which is REQUIRED so that
+# "widen the same canary" is the natural operation and advancing a rollout can
+# never silently promote unreviewed code that landed after the canary. The
+# literal value `latest` is an explicit opt-in for the "promote whatever was
+# uploaded last" case (e.g. right after a merge); it is never the default.
 on:
   push:
     inputs:
       promote_percent:
         description: >-
-          Promote the latest uploaded cloud version to this percentage of
+          Promote a cloud version to this percentage of traffic (1-100).
-          traffic (1-100). Leave empty to upload only (no promotion).
+          Leave empty to do nothing (this workflow only uploads on push to
+          main; a manual run is for promotion only).
+        required: false
+        default: ""
+      promote_version:
+        description: >-
+          The cloud version id to promote (from `wrangler versions list`).
+          Required when promote_percent is set. Use the literal `latest` to
+          promote the most recently uploaded version (opt-in; only correct
+          right after the intended merge, never for advancing an existing
+          canary).
         required: false
         default: ""
   cancel-in-progress: false
 jobs:
+  # Runs only on the upload path (push to main). A promote-only
+  # workflow_dispatch must not re-run migrations.
   migrate:
     name: Migrate database
     runs-on: blacksmith-4vcpu-ubuntu-2404
     environment: production
+    if: github.event_name == 'push'
     steps:
       - uses: actions/checkout@v4
     runs-on: blacksmith-4vcpu-ubuntu-2404
     environment: production
     needs: migrate
+    if: github.event_name == 'push'
     steps:
       - uses: actions/checkout@v4
           CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
           CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
-  # Promote the LATEST uploaded cloud version to a chosen percentage of live
+  # Promote a specific, already-uploaded cloud version to a chosen percentage
-  # traffic. Gated: runs only on a manual `workflow_dispatch` that sets
+  # of live traffic. DECOUPLED from migrate/upload: it does NOT `needs` them and
-  # `promote_percent`. A promotion moves a Durable Object's version and so
+  # runs only on a manual `workflow_dispatch` that sets `promote_percent`, so a
-  # restarts DOs whose sessions land on the newly promoted version -- keep
+  # promotion never builds a new version. The version to promote is named
-  # `promote_percent` small for the first step of a rollout, watch telemetry,
+  # explicitly by `promote_version` (required) so advancing an existing canary
-  # then re-run at a higher percentage. See apps/cloud/DEPLOY.md.
+  # widens THAT version rather than silently promoting whatever landed on main
+  # since. A promotion moves a Durable Object's version and so restarts DOs whose
+  # sessions land on the newly promoted version -- keep `promote_percent` small
+  # for the first step of a rollout, watch telemetry, then re-run at a higher
+  # percentage for the SAME `promote_version`. See apps/cloud/DEPLOY.md.
   promote-cloud:
     name: Promote cloud version
     runs-on: blacksmith-4vcpu-ubuntu-2404
     environment: production
-    needs: upload-cloud
     if: >-
       github.event_name == 'workflow_dispatch' &&
       inputs.promote_percent != ''
     steps:
+      # GitHub cannot conditionally require an input, so enforce it here: fail
+      # fast (before checkout/build) if promote_version is missing rather than
+      # letting the promote script error after the expensive steps.
+      - name: Require promote_version
+        if: inputs.promote_version == ''
+        run: |
+          echo "::error::promote_version is required when promote_percent is set. Pass a version id from 'wrangler versions list', or the literal 'latest'."
+          exit 1
       - uses: actions/checkout@v4
       - uses: oven-sh/setup-bun@v2
         env:
           VITE_PUBLIC_SENTRY_DSN: ${{ secrets.VITE_PUBLIC_SENTRY_DSN }}
-      - name: Promote latest uploaded version
+      - name: Promote cloud version
-        run: bun run scripts/promote-cloud.ts --percent "${PROMOTE_PERCENT}"
+        run: bun run scripts/promote-cloud.ts --percent "${PROMOTE_PERCENT}" --version "${PROMOTE_VERSION}"
         working-directory: apps/cloud
         env:
           PROMOTE_PERCENT: ${{ inputs.promote_percent }}
+          PROMOTE_VERSION: ${{ inputs.promote_version }}
           CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
           CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
+  # Marketing deploys on the normal merge-to-main path only. A promote-only
+  # workflow_dispatch targets the cloud worker and must not redeploy marketing.
   deploy-marketing:
     name: Deploy marketing
     runs-on: blacksmith-4vcpu-ubuntu-2404
     environment: production
+    if: github.event_name == 'push'
     steps:
       - uses: actions/checkout@v4

apps/cloud/DEPLOY.md+42−16

 ## The flow
-1. **Merge to main** runs `.github/workflows/deploy.yml`:
+1. **Merge to main** runs `.github/workflows/deploy.yml` (`push` trigger only):
    - `migrate` applies database migrations.
    - `upload-cloud` runs `wrangler versions upload`. This creates a new Worker
      **version** but routes **no traffic** to it. No DOs restart; live MCP
      sessions keep running. This is safe to do on every merge.
-2. **Promotion is a separate, deliberate step.** Re-run the Deploy workflow via
+   - `deploy-marketing` deploys the marketing worker (unchanged behaviour).
-   **workflow_dispatch** with a `promote_percent` input (1-100). The
+   - Note the uploaded **version id** from the `upload-cloud` job log (or run
-   `promote-cloud` job promotes the **latest uploaded** version to that
+     `wrangler versions list`): you promote it by id below.
-   percentage of traffic (`scripts/promote-cloud.ts`, which calls
+2. **Promotion is a separate, deliberate, DECOUPLED step.** Run the Deploy
-   `wrangler versions deploy`). For a canary, start small (e.g. 10), watch
+   workflow via **workflow_dispatch** with two inputs:
-   telemetry, then re-run at a higher percentage; run 100 for a full cutover.
+   - `promote_percent` (1-100): the traffic percentage for this step.
+   - `promote_version`: the **version id** to promote (required). Use the
+     literal `latest` only for "promote what I just merged"; for advancing an
+     existing canary always pass the **same version id** you canaried, so you
+     widen that reviewed version rather than something newer.
+   A promote-only dispatch runs **only** the `promote-cloud` job: it does **not**
+   re-run `migrate` or `upload-cloud`, so it never builds a new version. The job
+   promotes `promote_version` to `promote_percent` of traffic
+   (`scripts/promote-cloud.ts`, which calls `wrangler versions deploy`).
+   Canary walk for one version `V`:
+   - dispatch `promote_percent=10`, `promote_version=V` (10% on `V`);
+   - watch telemetry;
+   - dispatch `promote_percent=50`, `promote_version=V` (widen the **same** `V`);
+   - dispatch `promote_percent=100`, `promote_version=V` (full cutover).
+   Merges that land **after** you start a canary upload their own versions but do
+   **not** get promoted until you dispatch with **their** id: advancing `V` never
+   silently ships unreviewed code.
 ### Promoting from the CLI (manual, out of band)
-From `apps/cloud`, against the generated build config:
+From `apps/cloud`, against the generated build config. `--version` is required;
+it is either an explicit version id or the literal `latest`:
 ```bash
 bun run build
-# Canary: latest uploaded version at 10%, current-live at 90%.
+wrangler versions list -c dist/server/wrangler.json    # find the version id
-bun run scripts/promote-cloud.ts --percent 10
+# Canary: version <id> at 10%, current-live at 90%.
-# Advance:
+bun run scripts/promote-cloud.ts --percent 10 --version <id>
-bun run scripts/promote-cloud.ts --percent 50
+# Advance (same version id):
-# Full cutover:
+bun run scripts/promote-cloud.ts --percent 50 --version <id>
-bun run scripts/promote-cloud.ts --percent 100
+# Full cutover (same version id):
+bun run scripts/promote-cloud.ts --percent 100 --version <id>
+# Or, right after a merge, promote the most recent upload explicitly:
+bun run scripts/promote-cloud.ts --percent 10 --version latest
 ```
-The script sorts versions by version number, promotes the newest, and holds the
+The script promotes exactly the version you name (`latest` resolves to the
-remainder of a partial promotion on the current-live version.
+highest version number) and holds the remainder of a partial promotion on the
+current-live version. It refuses to run unless the resolved wrangler config
+targets the `executor-cloud` worker (set `PROMOTE_CLOUD_ALLOW_NAME` to override
+for a non-production worker such as the deploy lab).
 ## Session version-affinity (zone ruleset, MUST be configured manually)

apps/cloud/scripts/promote-cloud.ts+91−28

@@ -1,21 +1,32 @@
1	/* oxlint-disable executor/no-error-constructor, executor/no-try-catch-or-throw -- boundary: out-of-band deploy-promotion CLI */	1	/* oxlint-disable executor/no-error-constructor, executor/no-try-catch-or-throw -- boundary: out-of-band deploy-promotion CLI */
2		2
3	/**	3	/**
4	* Promote the LATEST uploaded executor-cloud Worker version to a chosen	4	* Promote a specific, already-uploaded executor-cloud Worker version to a
5	* percentage of live traffic, using Cloudflare's staged gradual-deployment	5	* chosen percentage of live traffic, using Cloudflare's staged
6	* model (`wrangler versions deploy`).	6	* gradual-deployment model (`wrangler versions deploy`).
7	*	7	*
8	* This is the second half of the upload -> staged-promotion deploy flow	8	* This is the second half of the upload -> staged-promotion deploy flow
9	* (see apps/cloud/DEPLOY.md and .github/workflows/deploy.yml). Merges upload a	9	* (see apps/cloud/DEPLOY.md and .github/workflows/deploy.yml). Merges upload a
10	* version but route no traffic to it; this script promotes it deliberately.	10	* version but route no traffic to it; this script promotes it deliberately.
11	*	11	*
12	* Behaviour:	12	* The version to promote is chosen EXPLICITLY, never inferred silently:
13	* --percent 100 promote the latest uploaded version to 100% (full cutover).	13	* --version <id> promote exactly that version id (from
14	* --percent 10 split traffic 10% latest / 90% current-live version, for a	14	* `wrangler versions list`). This is what advances an
15	* canary step. Re-run at a higher percentage to advance the	15	* existing canary: re-run at a higher --percent for the SAME
16	* rollout; the DOs whose sessions land on the newly promoted	16	* --version so you widen the version you already reviewed,
17	* version restart, so keep early steps small and watch	17	* not whatever landed on main since.
18	* telemetry between steps.	18	* --version latest promote the most recently uploaded version. This is an
		19	* explicit opt-in for "promote what I just merged"; it is
		20	* NEVER the default, so advancing a rollout can never
		21	* accidentally promote unreviewed code.
		22	*
		23	* Percentage:
		24	* --percent 100 promote the chosen version to 100% (full cutover).
		25	* --percent 10 split traffic 10% chosen / 90% current-live version, for a
		26	* canary step. Re-run at a higher percentage (same version)
		27	* to advance; the DOs whose sessions land on the newly
		28	* promoted version restart, so keep early steps small and
		29	* watch telemetry between steps.
19	*	30	*
20	* IMPORTANT: a promotion that changes a Durable Object's assigned version	31	* IMPORTANT: a promotion that changes a Durable Object's assigned version
21	* restarts that DO (validated empirically, DEPLOY.md experiments 3b/5). Stage	32	* restarts that DO (validated empirically, DEPLOY.md experiments 3b/5). Stage
@@ -25,11 +36,36 @@ · 3 unmodified lines
25	* in the CI env). Config is the generated dist config the build produced.	36	* in the CI env). Config is the generated dist config the build produced.
26	*/	37	*/
27	import { spawnSync } from "node:child_process";	38	import { spawnSync } from "node:child_process";
		39	import { readFileSync } from "node:fs";
28		40
29	// The generated dist config the build produced. Overridable via	41	// The generated dist config the build produced. Overridable via
30	// PROMOTE_CLOUD_CONFIG for local/lab testing against a non-prod worker.	42	// PROMOTE_CLOUD_CONFIG for local/lab testing against a non-prod worker.
31	const CONFIG = process.env.PROMOTE_CLOUD_CONFIG ?? "dist/server/wrangler.json";	43	const CONFIG = process.env.PROMOTE_CLOUD_CONFIG ?? "dist/server/wrangler.json";
32		44
		45	// Target-worker guard: refuse to run against anything but executor-cloud unless
		46	// the operator explicitly names the worker they expect via
		47	// PROMOTE_CLOUD_ALLOW_NAME (for the deploy lab / a non-prod worker). A promotion
		48	// routes live traffic and restarts DOs, so a misconfigured config that points at
		49	// the wrong worker must fail loudly, not promote the wrong thing. `--yes` does
		50	// NOT bypass this: it only skips wrangler's own interactive confirmation.
		51	const EXPECTED_WORKER_NAME = process.env.PROMOTE_CLOUD_ALLOW_NAME ?? "executor-cloud";
		52	{
		53	let configName: unknown;
		54	try {
		55	configName = (JSON.parse(readFileSync(CONFIG, "utf8")) as { name?: unknown }).name;
		56	} catch (error) {
		57	throw new Error(
		58	`could not read wrangler config ${CONFIG} to verify the target worker: ${String(error)}`,
		59	);
		60	}
		61	if (configName !== EXPECTED_WORKER_NAME) {
		62	throw new Error(
		63	`refusing to promote: wrangler config ${CONFIG} targets worker ${JSON.stringify(configName)}, expected ${JSON.stringify(EXPECTED_WORKER_NAME)}. ` +
		64	`Set PROMOTE_CLOUD_ALLOW_NAME to override for a non-production worker.`,
		65	);
		66	}
		67	}
		68
33	const args = process.argv.slice(2);	69	const args = process.argv.slice(2);
34	const argValue = (name: string): string \| undefined => {	70	const argValue = (name: string): string \| undefined => {
35	const i = args.indexOf(name);	71	const i = args.indexOf(name);
@@ -45,6 +81,16 @@ if (!Number.isInteger(percent) \|\| percent < 1 \|\| percent > 100) { · 9 unmodified lines
45	throw new Error(`--percent must be an integer 1-100, got: ${percentRaw}`);	81	throw new Error(`--percent must be an integer 1-100, got: ${percentRaw}`);
46	}	82	}
47		83
		84	// The version to promote is required and explicit: an id, or the literal
		85	// `latest` to opt in to "promote the most recently uploaded version".
		86	const versionArg = argValue("--version") ?? process.env.PROMOTE_VERSION;
		87	if (!versionArg) {
		88	throw new Error(
		89	"--version is required: pass an explicit version id (from `wrangler versions list`), " +
		90	"or the literal `latest` to promote the most recently uploaded version.",
		91	);
		92	}
		93
48	const wrangler = (wranglerArgs: readonly string[]): string => {	94	const wrangler = (wranglerArgs: readonly string[]): string => {
49	const result = spawnSync("bun", ["run", "wrangler", ...wranglerArgs], {	95	const result = spawnSync("bun", ["run", "wrangler", ...wranglerArgs], {
50	encoding: "utf8",	96	encoding: "utf8",
@@ -67,21 +113,38 @@ type DeploymentStatus = { · 16 unmodified lines
67	readonly versions: ReadonlyArray<{ readonly version_id: string; readonly percentage: number }>;	113	readonly versions: ReadonlyArray<{ readonly version_id: string; readonly percentage: number }>;
68	};	114	};
69		115
70	// `versions list --json` order is not guaranteed, so sort explicitly by
71	// version number (monotonic per upload) descending to get the newest.
72	const versions = JSON.parse(	116	const versions = JSON.parse(
73	wrangler(["versions", "list", "-c", CONFIG, "--json"]),	117	wrangler(["versions", "list", "-c", CONFIG, "--json"]),
74	) as VersionListEntry[];	118	) as VersionListEntry[];
75	if (versions.length === 0) {	119	if (versions.length === 0) {
76	throw new Error("no uploaded versions found");	120	throw new Error("no uploaded versions found");
77	}	121	}
78	const sorted = [...versions].sort((a, b) => {	122
79	if (a.number !== b.number) return b.number - a.number;	123	// Resolve the target version EXPLICITLY. `latest` is an opt-in that sorts by
80	const at = a.metadata?.created_on ?? "";	124	// version number (monotonic per upload) descending, since `versions list`
81	const bt = b.metadata?.created_on ?? "";	125	// order is not guaranteed. Any other value is treated as a version id and must
82	return bt.localeCompare(at);	126	// match an uploaded version exactly, so a typo or a since-deleted id fails
83	});	127	// loudly rather than promoting the wrong version.
84	const latest = sorted[0];	128	let target: VersionListEntry;
		129	if (versionArg === "latest") {
		130	const sorted = [...versions].sort((a, b) => {
		131	if (a.number !== b.number) return b.number - a.number;
		132	const at = a.metadata?.created_on ?? "";
		133	const bt = b.metadata?.created_on ?? "";
		134	return bt.localeCompare(at);
		135	});
		136	target = sorted[0];
		137	} else {
		138	const found = versions.find((v) => v.id === versionArg);
		139	if (!found) {
		140	throw new Error(
		141	`version ${versionArg} not found among uploaded versions. ` +
		142	`Run \`wrangler versions list -c ${CONFIG}\` to see valid ids, ` +
		143	"or pass `latest` to promote the most recent upload.",
		144	);
		145	}
		146	target = found;
		147	}
85		148
86	// The version(s) currently serving live traffic, so a partial promotion holds	149	// The version(s) currently serving live traffic, so a partial promotion holds
87	// the remainder on the current-live version rather than an arbitrary one.	150	// the remainder on the current-live version rather than an arbitrary one.
@@ -90,24 +153,24 @@ const status = JSON.parse( · 2 unmodified lines
90	) as DeploymentStatus;	153	) as DeploymentStatus;
91	// The version to hold the remainder of a partial promotion: the current-live	154	// The version to hold the remainder of a partial promotion: the current-live
92	// version carrying the most traffic that is NOT the one we are promoting. If	155	// version carrying the most traffic that is NOT the one we are promoting. If
93	// the only current version IS the latest, there is nothing to split against.	156	// the only current version IS the target, there is nothing to split against.
94	const currentLive = [...status.versions]	157	const currentLive = [...status.versions]
95	.filter((v) => v.version_id !== latest.id)	158	.filter((v) => v.version_id !== target.id)
96	.sort((a, b) => b.percentage - a.percentage)[0]?.version_id;	159	.sort((a, b) => b.percentage - a.percentage)[0]?.version_id;
97		160
98	if (latest.id === currentLive && percent === 100) {	161	if (target.id === currentLive && percent === 100) {
99	console.log(`Latest version ${latest.id} is already live at 100%; nothing to promote.`);	162	console.log(`Version ${target.id} is already live at 100%; nothing to promote.`);
100	process.exit(0);	163	process.exit(0);
101	}	164	}
102		165
103	const deployArgs = ["versions", "deploy", "-c", CONFIG, "--yes"];	166	const deployArgs = ["versions", "deploy", "-c", CONFIG, "--yes"];
104	if (percent === 100 \|\| !currentLive \|\| currentLive === latest.id) {	167	if (percent === 100 \|\| !currentLive \|\| currentLive === target.id) {
105	deployArgs.push(`${latest.id}@100%`);	168	deployArgs.push(`${target.id}@100%`);
106	} else {	169	} else {
107	// Staged split: latest@percent, current-live@remainder.	170	// Staged split: target@percent, current-live@remainder.
108	deployArgs.push(`${latest.id}@${percent}%`, `${currentLive}@${100 - percent}%`);	171	deployArgs.push(`${target.id}@${percent}%`, `${currentLive}@${100 - percent}%`);
109	}	172	}
110		173
111	console.log(`Promoting ${latest.id} (version #${latest.number}) at ${percent}%...`);	174	console.log(`Promoting ${target.id} (version #${target.number}) at ${percent}%...`);
112	console.log(wrangler(deployArgs));	175	console.log(wrangler(deployArgs));
113	console.log("Done. Monitor telemetry, then re-run at a higher percentage to advance the rollout.");	176	console.log("Done. Monitor telemetry, then re-run at a higher percentage to advance the rollout.");