History

Promote a named cloud version, decoupled from upload/migrate

A promote-only workflow_dispatch previously re-ran migrate and upload-cloud (building a fresh, un-canaried version) and then promoted whatever was newest, so advancing a canary could ship code that landed on main after the canary started. Decouple promotion: the promote-cloud job no longer needs upload, and migrate/upload/deploy-marketing run on push only. The version to promote is now an explicit promote_version input (a version id, or the literal latest as an opt-in), and promote-cloud.ts promotes exactly that version. Also guard the script against running with a wrangler config whose worker name is not executor-cloud.

Rhys Sullivan committed 4 months ago

Commit 5d11ea51 parent 4b260753 files · +182 −58Browse files

Showing the changes to apps/cloud/scripts (1 of 3 files). Show all

Unified · Split

apps/cloud/scripts/promote-cloud.ts+91−28
@@ -1,21 +1,32 @@
1/* oxlint-disable executor/no-error-constructor, executor/no-try-catch-or-throw -- boundary: out-of-band deploy-promotion CLI */1/* oxlint-disable executor/no-error-constructor, executor/no-try-catch-or-throw -- boundary: out-of-band deploy-promotion CLI */
2 2
3/**3/**
4 * Promote the LATEST uploaded executor-cloud Worker version to a chosen4 * Promote a specific, already-uploaded executor-cloud Worker version to a
5 * percentage of live traffic, using Cloudflare's staged gradual-deployment5 * chosen percentage of live traffic, using Cloudflare's staged
6 * model (`wrangler versions deploy`).6 * gradual-deployment model (`wrangler versions deploy`).
7 *7 *
8 * This is the second half of the upload -> staged-promotion deploy flow8 * This is the second half of the upload -> staged-promotion deploy flow
9 * (see apps/cloud/DEPLOY.md and .github/workflows/deploy.yml). Merges upload a9 * (see apps/cloud/DEPLOY.md and .github/workflows/deploy.yml). Merges upload a
10 * version but route no traffic to it; this script promotes it deliberately.10 * version but route no traffic to it; this script promotes it deliberately.
11 *11 *
12 * Behaviour:12 * The version to promote is chosen EXPLICITLY, never inferred silently:
13 * --percent 100 promote the latest uploaded version to 100% (full cutover).13 * --version <id> promote exactly that version id (from
14 * --percent 10 split traffic 10% latest / 90% current-live version, for a14 * `wrangler versions list`). This is what advances an
15 * canary step. Re-run at a higher percentage to advance the15 * existing canary: re-run at a higher --percent for the SAME
16 * rollout; the DOs whose sessions land on the newly promoted16 * --version so you widen the version you already reviewed,
17 * version restart, so keep early steps small and watch17 * not whatever landed on main since.
18 * telemetry between steps.18 * --version latest promote the most recently uploaded version. This is an
19 * explicit opt-in for "promote what I just merged"; it is
20 * NEVER the default, so advancing a rollout can never
21 * accidentally promote unreviewed code.
22 *
23 * Percentage:
24 * --percent 100 promote the chosen version to 100% (full cutover).
25 * --percent 10 split traffic 10% chosen / 90% current-live version, for a
26 * canary step. Re-run at a higher percentage (same version)
27 * to advance; the DOs whose sessions land on the newly
28 * promoted version restart, so keep early steps small and
29 * watch telemetry between steps.
19 *30 *
20 * IMPORTANT: a promotion that changes a Durable Object's assigned version31 * IMPORTANT: a promotion that changes a Durable Object's assigned version
21 * restarts that DO (validated empirically, DEPLOY.md experiments 3b/5). Stage32 * restarts that DO (validated empirically, DEPLOY.md experiments 3b/5). Stage
@@ -25,11 +36,36 @@ · 3 unmodified lines
25 * in the CI env). Config is the generated dist config the build produced.36 * in the CI env). Config is the generated dist config the build produced.
26 */37 */
27import { spawnSync } from "node:child_process";38import { spawnSync } from "node:child_process";
39import { readFileSync } from "node:fs";
28 40
29// The generated dist config the build produced. Overridable via41// The generated dist config the build produced. Overridable via
30// PROMOTE_CLOUD_CONFIG for local/lab testing against a non-prod worker.42// PROMOTE_CLOUD_CONFIG for local/lab testing against a non-prod worker.
31const CONFIG = process.env.PROMOTE_CLOUD_CONFIG ?? "dist/server/wrangler.json";43const CONFIG = process.env.PROMOTE_CLOUD_CONFIG ?? "dist/server/wrangler.json";
32 44
45// Target-worker guard: refuse to run against anything but executor-cloud unless
46// the operator explicitly names the worker they expect via
47// PROMOTE_CLOUD_ALLOW_NAME (for the deploy lab / a non-prod worker). A promotion
48// routes live traffic and restarts DOs, so a misconfigured config that points at
49// the wrong worker must fail loudly, not promote the wrong thing. `--yes` does
50// NOT bypass this: it only skips wrangler's own interactive confirmation.
51const EXPECTED_WORKER_NAME = process.env.PROMOTE_CLOUD_ALLOW_NAME ?? "executor-cloud";
52{
53 let configName: unknown;
54 try {
55 configName = (JSON.parse(readFileSync(CONFIG, "utf8")) as { name?: unknown }).name;
56 } catch (error) {
57 throw new Error(
58 `could not read wrangler config ${CONFIG} to verify the target worker: ${String(error)}`,
59 );
60 }
61 if (configName !== EXPECTED_WORKER_NAME) {
62 throw new Error(
63 `refusing to promote: wrangler config ${CONFIG} targets worker ${JSON.stringify(configName)}, expected ${JSON.stringify(EXPECTED_WORKER_NAME)}. ` +
64 `Set PROMOTE_CLOUD_ALLOW_NAME to override for a non-production worker.`,
65 );
66 }
67}
68
33const args = process.argv.slice(2);69const args = process.argv.slice(2);
34const argValue = (name: string): string | undefined => {70const argValue = (name: string): string | undefined => {
35 const i = args.indexOf(name);71 const i = args.indexOf(name);
@@ -45,6 +81,16 @@ if (!Number.isInteger(percent) || percent < 1 || percent > 100) { · 9 unmodified lines
45 throw new Error(`--percent must be an integer 1-100, got: ${percentRaw}`);81 throw new Error(`--percent must be an integer 1-100, got: ${percentRaw}`);
46}82}
47 83
84// The version to promote is required and explicit: an id, or the literal
85// `latest` to opt in to "promote the most recently uploaded version".
86const versionArg = argValue("--version") ?? process.env.PROMOTE_VERSION;
87if (!versionArg) {
88 throw new Error(
89 "--version is required: pass an explicit version id (from `wrangler versions list`), " +
90 "or the literal `latest` to promote the most recently uploaded version.",
91 );
92}
93
48const wrangler = (wranglerArgs: readonly string[]): string => {94const wrangler = (wranglerArgs: readonly string[]): string => {
49 const result = spawnSync("bun", ["run", "wrangler", ...wranglerArgs], {95 const result = spawnSync("bun", ["run", "wrangler", ...wranglerArgs], {
50 encoding: "utf8",96 encoding: "utf8",
@@ -67,21 +113,38 @@ type DeploymentStatus = { · 16 unmodified lines
67 readonly versions: ReadonlyArray<{ readonly version_id: string; readonly percentage: number }>;113 readonly versions: ReadonlyArray<{ readonly version_id: string; readonly percentage: number }>;
68};114};
69 115
70// `versions list --json` order is not guaranteed, so sort explicitly by
71// version number (monotonic per upload) descending to get the newest.
72const versions = JSON.parse(116const versions = JSON.parse(
73 wrangler(["versions", "list", "-c", CONFIG, "--json"]),117 wrangler(["versions", "list", "-c", CONFIG, "--json"]),
74) as VersionListEntry[];118) as VersionListEntry[];
75if (versions.length === 0) {119if (versions.length === 0) {
76 throw new Error("no uploaded versions found");120 throw new Error("no uploaded versions found");
77}121}
78const sorted = [...versions].sort((a, b) => {122
79 if (a.number !== b.number) return b.number - a.number;123// Resolve the target version EXPLICITLY. `latest` is an opt-in that sorts by
80 const at = a.metadata?.created_on ?? "";124// version number (monotonic per upload) descending, since `versions list`
81 const bt = b.metadata?.created_on ?? "";125// order is not guaranteed. Any other value is treated as a version id and must
82 return bt.localeCompare(at);126// match an uploaded version exactly, so a typo or a since-deleted id fails
83});127// loudly rather than promoting the wrong version.
84const latest = sorted[0];128let target: VersionListEntry;
129if (versionArg === "latest") {
130 const sorted = [...versions].sort((a, b) => {
131 if (a.number !== b.number) return b.number - a.number;
132 const at = a.metadata?.created_on ?? "";
133 const bt = b.metadata?.created_on ?? "";
134 return bt.localeCompare(at);
135 });
136 target = sorted[0];
137} else {
138 const found = versions.find((v) => v.id === versionArg);
139 if (!found) {
140 throw new Error(
141 `version ${versionArg} not found among uploaded versions. ` +
142 `Run \`wrangler versions list -c ${CONFIG}\` to see valid ids, ` +
143 "or pass `latest` to promote the most recent upload.",
144 );
145 }
146 target = found;
147}
85 148
86// The version(s) currently serving live traffic, so a partial promotion holds149// The version(s) currently serving live traffic, so a partial promotion holds
87// the remainder on the current-live version rather than an arbitrary one.150// the remainder on the current-live version rather than an arbitrary one.
@@ -90,24 +153,24 @@ const status = JSON.parse( · 2 unmodified lines
90) as DeploymentStatus;153) as DeploymentStatus;
91// The version to hold the remainder of a partial promotion: the current-live154// The version to hold the remainder of a partial promotion: the current-live
92// version carrying the most traffic that is NOT the one we are promoting. If155// version carrying the most traffic that is NOT the one we are promoting. If
93// the only current version IS the latest, there is nothing to split against.156// the only current version IS the target, there is nothing to split against.
94const currentLive = [...status.versions]157const currentLive = [...status.versions]
95 .filter((v) => v.version_id !== latest.id)158 .filter((v) => v.version_id !== target.id)
96 .sort((a, b) => b.percentage - a.percentage)[0]?.version_id;159 .sort((a, b) => b.percentage - a.percentage)[0]?.version_id;
97 160
98if (latest.id === currentLive && percent === 100) {161if (target.id === currentLive && percent === 100) {
99 console.log(`Latest version ${latest.id} is already live at 100%; nothing to promote.`);162 console.log(`Version ${target.id} is already live at 100%; nothing to promote.`);
100 process.exit(0);163 process.exit(0);
101}164}
102 165
103const deployArgs = ["versions", "deploy", "-c", CONFIG, "--yes"];166const deployArgs = ["versions", "deploy", "-c", CONFIG, "--yes"];
104if (percent === 100 || !currentLive || currentLive === latest.id) {167if (percent === 100 || !currentLive || currentLive === target.id) {
105 deployArgs.push(`${latest.id}@100%`);168 deployArgs.push(`${target.id}@100%`);
106} else {169} else {
107 // Staged split: latest@percent, current-live@remainder.170 // Staged split: target@percent, current-live@remainder.
108 deployArgs.push(`${latest.id}@${percent}%`, `${currentLive}@${100 - percent}%`);171 deployArgs.push(`${target.id}@${percent}%`, `${currentLive}@${100 - percent}%`);
109}172}
110 173
111console.log(`Promoting ${latest.id} (version #${latest.number}) at ${percent}%...`);174console.log(`Promoting ${target.id} (version #${target.number}) at ${percent}%...`);
112console.log(wrangler(deployArgs));175console.log(wrangler(deployArgs));
113console.log("Done. Monitor telemetry, then re-run at a higher percentage to advance the rollout.");176console.log("Done. Monitor telemetry, then re-run at a higher percentage to advance the rollout.");