History

Enforce execution limits with a balance gate and per-org rate-limit backstop (#1274)

* Add execution balance check to billing service * Add pre-execution balance gate for org execution limits Check the org's Autumn execution balance before execute/executeWithPause (never resume, so paused executions can always complete). Blocked orgs get a descriptive ExecuteResult.error instead of running. Fails open on any billing error or a 2s timeout, and caches per-org outcomes for 60s. * Add per-org execution rate-limit backstop 1000 execute calls per org per hour, fixed window, counted in a minimal per-org Durable Object (EXECUTION_RATE_LIMITER, one instance per org via idFromName). Independent of billing so runaway automation is caught even when Autumn is down; fails open if the counter DO errors or is slow. The DO stores a single {windowId, count} record and purges itself by alarm after two idle windows. * Wire execution guards into the metered engine decorator Order: rate-limit backstop first (cheap counter), then the balance gate, then usage tracking. Guards wrap outside the tracker so a blocked execution is neither run nor tracked. Covers both planes (HTTP executor plane and MCP session DO) since both build engines through this layer. * Test the execution balance gate and rate-limit backstop Covers allow/block, the typed errors, fail-open on billing errors and timeouts (asserting the check was attempted AND the execution ran), the 60s per-org outcome cache (allowed and blocked cached, errors never), window rollover, per-org isolation, and that resume is never gated. * Use typed test errors and instanceof assertions in gate tests * Replace execution-limit unit tests with e2e coverage Delete the balance-gate and rate-limit unit tests: the guards are now exercised end to end against the real workerd + Autumn emulator, which is a truer test of behaviour that mattered here (blocked executions not reaching the meter, failing open on billing errors). Drop the test-only affordances the unit tests were the only callers of: the gate's `timeoutMs` option knob and both limiters' `check` diagnostic methods. Move the two blocked-execution messages into a small Cloudflare-free module so the e2e suite can import them as assertion ground truth without pulling the guards' worker-runtime dependencies into its typecheck. * Make the per-org execution rate limit env-configurable Add an optional EXECUTION_RATE_LIMIT_PER_HOUR override, parsed as a positive integer and falling back to the 1000/hour default when unset or unparseable. Production leaves it unset; it exists so e2e can drive the abuse backstop with a handful of real executions instead of a thousand. The e2e cloud dev server sets it to 3, reaching the worker the same way ALLOW_LOCAL_NETWORK does. * Add Autumn e2e helpers for fault injection and balance exhaustion Bump the emulator to 0.13.2 for its balances.check support and one-shot fault injection, and extend the Autumn surface to use them: armFault/clearFaults drive the emulator's fault control plane, exhaustExecutions burns an org's whole included allotment in one track (the amount read from the plan seed, never hardcoded), and ledgerFor reads the request ledger so a scenario can prove a faulted balances.check was actually attempted. * Add e2e scenarios for the pre-execution limit guards Four cloud scenarios drive a real MCP client against the workerd + session-DO topology and read Autumn's ledger as ground truth: an org at its cap is blocked before the code runs and not metered; a balance-check 500 and a balance-check timeout each fail open and still run the execution (with the faulted check visible in the ledger as proof the gate consulted Autumn); and the rate-limit backstop blocks the execution past the hourly cap without metering it. Faults are one-shot and cleared in finalizers, and each scenario uses a fresh org. * Raise the e2e execution rate cap above other scenarios' usage The e2e worker's per-org hourly cap of 3 was low enough that ordinary scenarios (toolkits-mcp, no-auth-connection, connect-handoff) tripped the backstop mid-test once the suite ran them against the shared worker. Lift it to 20 (still exhaustible by the backstop scenario) and move the value into one shared constant consumed by both the boot env and the test.

Rhys Sullivan committed 4 months ago

Commit cfd02cf1 parent 718ffd215 files · +855 −13Browse files

Unified · Split

15 files
  1. apps/cloud/src/engine/execution-gate.ts +185 −0
  2. apps/cloud/src/engine/execution-limit-messages.ts +11 −0
  3. apps/cloud/src/engine/execution-rate-limit.ts +220 −0
  4. apps/cloud/src/engine/execution-stack-metered.ts +30 −9
  5. apps/cloud/src/env-augment.d.ts +14 −0
  6. apps/cloud/src/extensions/billing/service.ts +14 −1
  7. apps/cloud/src/server.ts +5 −0
  8. apps/cloud/wrangler.jsonc +11 −0
  9. bun.lock +2 −2
  10. e2e/cloud/mcp-execution-limits.test.ts +200 −0
  11. e2e/package.json +1 −1
  12. e2e/setup/autumn-plans.ts +18 −0
  13. e2e/setup/cloud.boot.ts +7 −0
  14. e2e/setup/execution-limits.ts +14 −0
  15. e2e/src/surfaces/autumn.ts +123 −0
apps/cloud/src/engine/execution-gate.ts+185−0
@@ -0,0 +1,185 @@
1+// ---------------------------------------------------------------------------
2+// Pre-execution balance gate — blocks new executions once an organization has
3+// used up its plan's included executions.
4+//
5+// Usage is tracked to Autumn after every execution (execution-usage.ts), but
6+// nothing ever CHECKED the balance before running, so a free-tier org could run
7+// unbounded executions past its quota. This gate consults
8+// `AutumnService.checkExecutionBalance` before `execute` / `executeWithPause`.
9+// `resume` is never gated: a paused execution already consumed its quota slot
10+// when it started, and blocking resume would strand approved work forever.
11+//
12+// FAIL OPEN is a hard requirement: any Autumn error, timeout, or missing
13+// customer/feature allows the execution (logged + Sentry, mirroring
14+// `trackExecution`'s reporting). Autumn being down must never block
15+// executions or add meaningful latency, so the check is bounded by a short
16+// timeout and its outcome is cached per organization.
17+//
18+// Error surfacing: the gate fails internally with the typed
19+// `ExecutionLimitReachedError`, and the engine seam folds it into the
20+// descriptive `ExecuteResult.error` channel. That is the codebase's mechanism
21+// for getting a domain failure's message to the MCP client verbatim — engine
22+// error-channel failures are deliberately rendered opaque by the MCP host
23+// ("Internal tool error [correlation id]"), exactly like the runtimes fold
24+// `CodeCompilationError` / `SandboxRuntimeError` into `ExecuteResult.error`.
25+// ---------------------------------------------------------------------------
26+
27+import * as Sentry from "@sentry/cloudflare";
28+import { Data, Effect } from "effect";
29+import type * as Cause from "effect/Cause";
30+
31+import type { ExecutionEngine, ExecutionResult } from "@executor-js/execution";
32+
33+import { EXECUTION_LIMIT_BLOCKED_MESSAGE } from "./execution-limit-messages";
34+
35+// The engine's completed-result payload (`ExecuteResult` in codemode-core),
36+// derived from the public execution types so this app package doesn't need a
37+// direct dependency on the kernel package that declares it.
38+type EngineExecuteResult = Extract<ExecutionResult, { status: "completed" }>["result"];
39+
40+// One check per org per minute is fresh enough for a monthly quota; both
41+// allowed AND blocked outcomes are cached (errors never are).
42+const BALANCE_CACHE_TTL_MS = 60_000;
43+// Autumn slower than this => fail open rather than stall a user-facing
44+// execution behind the billing provider.
45+const BALANCE_CHECK_TIMEOUT_MS = 2_000;
46+// Sweep guard so a long-lived worker isolate serving many orgs can't grow the
47+// cache map unbounded.
48+const BALANCE_CACHE_MAX_ENTRIES = 10_000;
49+
50+export { EXECUTION_LIMIT_BLOCKED_MESSAGE };
51+
52+export class ExecutionLimitReachedError extends Data.TaggedError("ExecutionLimitReachedError")<{
53+ readonly organizationId: string;
54+ readonly message: string;
55+}> {}
56+
57+/** Internal sentinel for a balance check that exceeded its time budget. */
58+class GateCheckTimeoutError extends Data.TaggedError("GateCheckTimeoutError")<{
59+ readonly timeoutMs: number;
60+}> {}
61+
62+// ---------------------------------------------------------------------------
63+// Shared gate seam — used by this gate and the rate-limit backstop.
64+// ---------------------------------------------------------------------------
65+
66+export type GateDecision =
67+ | { readonly blocked: false }
68+ | { readonly blocked: true; readonly error: { readonly message: string } };
69+
70+/**
71+ * Wrap an engine so `decide` runs before `execute` / `executeWithPause`. A
72+ * blocked decision short-circuits to a descriptive `ExecuteResult.error`
73+ * (which `formatExecuteResult` renders as a clean `isError` MCP tool result)
74+ * WITHOUT invoking the inner engine — so a blocked execution is neither run
75+ * nor usage-tracked. `resume` and all read-only members pass through
76+ * untouched: paused executions must always be able to complete.
77+ */
78+export const withPreExecutionGate = <E extends Cause.YieldableError>(
79+ engine: ExecutionEngine<E>,
80+ decide: Effect.Effect<GateDecision>,
81+): ExecutionEngine<E> => ({
82+ execute: (code, options) =>
83+ Effect.flatMap(
84+ decide,
85+ (decision): Effect.Effect<EngineExecuteResult, E> =>
86+ decision.blocked
87+ ? Effect.succeed({ result: null, error: decision.error.message })
88+ : engine.execute(code, options),
89+ ),
90+ executeWithPause: (code, options) =>
91+ Effect.flatMap(
92+ decide,
93+ (decision): Effect.Effect<ExecutionResult, E> =>
94+ decision.blocked
95+ ? Effect.succeed({
96+ status: "completed",
97+ result: { result: null, error: decision.error.message },
98+ })
99+ : engine.executeWithPause(code, options),
100+ ),
101+ // resume is never gated: paused executions must be able to complete.
102+ resume: (executionId, response) => engine.resume(executionId, response),
103+ getPausedExecution: (executionId) => engine.getPausedExecution(executionId),
104+ pausedExecutionCount: () => engine.pausedExecutionCount(),
105+ hasPausedExecutions: () => engine.hasPausedExecutions(),
106+ getDescription: engine.getDescription,
107+});
108+
109+// ---------------------------------------------------------------------------
110+// Balance gate factory
111+// ---------------------------------------------------------------------------
112+
113+export type ExecutionBalanceCheck = (
114+ organizationId: string,
115+) => Effect.Effect<{ readonly allowed: boolean }, unknown>;
116+
117+/**
118+ * Build a balance gate around `checkBalance` (in production:
119+ * `AutumnService.checkExecutionBalance`). One gate instance holds one
120+ * per-organization outcome cache; the metered decorator layer creates a single
121+ * instance so all engines it decorates share it.
122+ */
123+export const makeExecutionLimitGate = (checkBalance: ExecutionBalanceCheck) => {
124+ const timeoutMs = BALANCE_CHECK_TIMEOUT_MS;
125+ const cache = new Map<string, { readonly allowed: boolean; readonly expiresAtMs: number }>();
126+
127+ const writeCache = (organizationId: string, allowed: boolean, nowMs: number): void => {
128+ if (cache.size >= BALANCE_CACHE_MAX_ENTRIES) {
129+ for (const [key, entry] of cache) {
130+ if (entry.expiresAtMs <= nowMs) cache.delete(key);
131+ }
132+ // Still saturated after dropping expired entries: reset rather than grow.
133+ if (cache.size >= BALANCE_CACHE_MAX_ENTRIES) cache.clear();
134+ }
135+ cache.set(organizationId, { allowed, expiresAtMs: nowMs + BALANCE_CACHE_TTL_MS });
136+ };
137+
138+ const toDecision = (organizationId: string, allowed: boolean): GateDecision =>
139+ allowed
140+ ? { blocked: false }
141+ : {
142+ blocked: true,
143+ error: new ExecutionLimitReachedError({
144+ organizationId,
145+ message: EXECUTION_LIMIT_BLOCKED_MESSAGE,
146+ }),
147+ };
148+
149+ const decide = (organizationId: string): Effect.Effect<GateDecision> =>
150+ Effect.suspend(() => {
151+ const nowMs = Date.now();
152+ const cached = cache.get(organizationId);
153+ if (cached && cached.expiresAtMs > nowMs) {
154+ return Effect.succeed(toDecision(organizationId, cached.allowed));
155+ }
156+ return checkBalance(organizationId).pipe(
157+ Effect.timeoutOrElse({
158+ duration: `${timeoutMs} millis`,
159+ orElse: () => Effect.fail(new GateCheckTimeoutError({ timeoutMs })),
160+ }),
161+ Effect.map(({ allowed }) => {
162+ writeCache(organizationId, allowed, nowMs);
163+ return toDecision(organizationId, allowed);
164+ }),
165+ // FAIL OPEN: Autumn errors, timeouts, and missing customers/features
166+ // must never block executions. Reported like `trackExecution` so a
167+ // billing outage still pages; the error outcome is never cached.
168+ Effect.catch((error: unknown) =>
169+ Effect.sync((): GateDecision => {
170+ console.warn("[billing] execution balance check failed open:", error);
171+ Sentry.captureException(error);
172+ return { blocked: false };
173+ }),
174+ ),
175+ );
176+ });
177+
178+ return {
179+ /** Wrap an engine so the balance gate runs before each new execution. */
180+ decorate: <E extends Cause.YieldableError>(
181+ organizationId: string,
182+ engine: ExecutionEngine<E>,
183+ ): ExecutionEngine<E> => withPreExecutionGate(engine, decide(organizationId)),
184+ };
185+};
apps/cloud/src/engine/execution-limit-messages.ts+11−0
@@ -0,0 +1,11 @@
1+// The user-facing messages a blocked execution surfaces to the MCP client, kept
2+// in their own Cloudflare-free module so the e2e suite can import them as
3+// ground truth without pulling the guards' `cloudflare:workers` dependencies
4+// into its (worker-types-free) typecheck. The guard modules re-export these, so
5+// production call sites are unchanged.
6+
7+export const EXECUTION_LIMIT_BLOCKED_MESSAGE =
8+ "Execution limit reached: your plan's included executions for this billing period are used up. Upgrade your plan or wait for the reset to continue.";
9+
10+export const RATE_LIMIT_BLOCKED_MESSAGE =
11+ "Rate limit exceeded: too many executions this hour. This is an abuse backstop — contact support if you hit this legitimately.";
apps/cloud/src/engine/execution-rate-limit.ts+220−0
@@ -0,0 +1,220 @@
1+// ---------------------------------------------------------------------------
2+// Per-org execution rate limit — an abuse backstop independent of billing.
3+//
4+// The balance gate (execution-gate.ts) depends on Autumn and fails open, so a
5+// billing outage plus runaway automation could still run unbounded executions.
6+// This limiter counts `execute` calls per organization in a fixed hourly
7+// window, backed by a minimal counter Durable Object (cross-session state:
8+// each MCP session lives in its own DO instance, so an in-memory counter
9+// would be per-session and trivially bypassed by opening more sessions).
10+//
11+// Like the balance gate it FAILS OPEN: an unreachable counter DO, a missing
12+// binding, or a slow call allows the execution (warn + Sentry). The backstop
13+// must never take executions down with it.
14+// ---------------------------------------------------------------------------
15+
16+import { DurableObject, env } from "cloudflare:workers";
17+import * as Sentry from "@sentry/cloudflare";
18+import { Data, Effect } from "effect";
19+import type * as Cause from "effect/Cause";
20+
21+import type { ExecutionEngine } from "@executor-js/execution";
22+
23+import { withPreExecutionGate, type GateDecision } from "./execution-gate";
24+import { RATE_LIMIT_BLOCKED_MESSAGE } from "./execution-limit-messages";
25+
26+// Fixed window: all executions in the same clock hour share one counter.
27+export const RATE_LIMIT_WINDOW_MS = 3_600_000;
28+// Calibration: the heaviest legitimate org runs ~1.1k executions per MONTH,
29+// so 1000 per HOUR is far above any human-driven usage and only trips on
30+// runaway automation (the incident this backstops: ~18k in 30 days would
31+// still pass, which is fine — that class of overrun is the balance gate's
32+// job; this catches tight loops).
33+export const EXECUTIONS_PER_ORG_PER_HOUR = 1000;
34+// Counter DO slower than this => fail open rather than stall executions.
35+const RATE_LIMIT_CHECK_TIMEOUT_MS = 2_000;
36+// The DO purges its storage this long after the last increment, so idle orgs
37+// cost nothing. Two windows: long enough that an active window never purges.
38+const COUNTER_PURGE_AFTER_MS = 2 * RATE_LIMIT_WINDOW_MS;
39+
40+export { RATE_LIMIT_BLOCKED_MESSAGE };
41+
42+export class ExecutionRateLimitExceededError extends Data.TaggedError(
43+ "ExecutionRateLimitExceededError",
44+)<{
45+ readonly organizationId: string;
46+ readonly message: string;
47+}> {}
48+
49+/** Internal sentinel for a counter call that exceeded its time budget. */
50+class RateLimitCheckTimeoutError extends Data.TaggedError("RateLimitCheckTimeoutError")<{
51+ readonly timeoutMs: number;
52+}> {}
53+
54+// ---------------------------------------------------------------------------
55+// Counter Durable Object — one instance per organization (idFromName(orgId)).
56+// Stores a single { windowId, count } record: an increment in a new window
57+// resets the count, so old windows never accumulate. An alarm purges storage
58+// after inactivity.
59+// ---------------------------------------------------------------------------
60+
61+const WINDOW_RECORD_KEY = "window";
62+
63+type WindowRecord = {
64+ readonly windowId: number;
65+ readonly count: number;
66+};
67+
68+export class ExecutionRateLimiterDO extends DurableObject {
69+ private readonly counterStorage: DurableObjectState["storage"];
70+
71+ constructor(ctx: DurableObjectState, doEnv: Env) {
72+ super(ctx, doEnv);
73+ // Kept on an own field (not just inherited `this.ctx`) so tests can run
74+ // the class against a fake storage under the `cloudflare:workers` stub.
75+ this.counterStorage = ctx.storage;
76+ }
77+
78+ /** Add one execution to `windowId`'s counter and return the new count. */
79+ async increment(windowId: number): Promise<number> {
80+ const stored = await this.counterStorage.get<WindowRecord>(WINDOW_RECORD_KEY);
81+ const count = stored && stored.windowId === windowId ? stored.count + 1 : 1;
82+ await this.counterStorage.put(WINDOW_RECORD_KEY, { windowId, count });
83+ await this.counterStorage.setAlarm(Date.now() + COUNTER_PURGE_AFTER_MS);
84+ return count;
85+ }
86+
87+ async alarm(): Promise<void> {
88+ await this.counterStorage.deleteAll();
89+ }
90+}
91+
92+// ---------------------------------------------------------------------------
93+// Client
94+// ---------------------------------------------------------------------------
95+
96+/** Count one execution for (organizationId, windowId); returns the new count. */
97+export type RateLimitIncrement = (
98+ organizationId: string,
99+ windowId: number,
100+) => Effect.Effect<number, unknown>;
101+
102+export type ExecutionRateLimiter = {
103+ readonly decorate: <E extends Cause.YieldableError>(
104+ organizationId: string,
105+ engine: ExecutionEngine<E>,
106+ ) => ExecutionEngine<E>;
107+};
108+
109+/**
110+ * Build a rate limiter around an increment function (in production: the
111+ * counter DO). `options.limit` is the per-org hourly cap (production sets it
112+ * from the env override in `makeCloudExecutionRateLimiter`); the rest tune the
113+ * window and time budget.
114+ */
115+export const makeExecutionRateLimiter = (
116+ increment: RateLimitIncrement,
117+ options?: {
118+ readonly limit?: number;
119+ readonly windowMs?: number;
120+ readonly timeoutMs?: number;
121+ readonly now?: () => number;
122+ },
123+): ExecutionRateLimiter => {
124+ const limit = options?.limit ?? EXECUTIONS_PER_ORG_PER_HOUR;
125+ const windowMs = options?.windowMs ?? RATE_LIMIT_WINDOW_MS;
126+ const timeoutMs = options?.timeoutMs ?? RATE_LIMIT_CHECK_TIMEOUT_MS;
127+ const now = options?.now ?? Date.now;
128+
129+ const decide = (organizationId: string): Effect.Effect<GateDecision> =>
130+ Effect.suspend(() => {
131+ const windowId = Math.floor(now() / windowMs);
132+ return increment(organizationId, windowId).pipe(
133+ Effect.timeoutOrElse({
134+ duration: `${timeoutMs} millis`,
135+ orElse: () => Effect.fail(new RateLimitCheckTimeoutError({ timeoutMs })),
136+ }),
137+ Effect.map(
138+ (count): GateDecision =>
139+ count > limit
140+ ? {
141+ blocked: true,
142+ error: new ExecutionRateLimitExceededError({
143+ organizationId,
144+ message: RATE_LIMIT_BLOCKED_MESSAGE,
145+ }),
146+ }
147+ : { blocked: false },
148+ ),
149+ // FAIL OPEN: the backstop must never block executions because its
150+ // counter is unreachable or slow.
151+ Effect.catch((error: unknown) =>
152+ Effect.sync((): GateDecision => {
153+ console.warn("[rate-limit] execution rate limit check failed open:", error);
154+ Sentry.captureException(error);
155+ return { blocked: false };
156+ }),
157+ ),
158+ );
159+ });
160+
161+ return {
162+ decorate: (organizationId, engine) => withPreExecutionGate(engine, decide(organizationId)),
163+ };
164+};
165+
166+// ---------------------------------------------------------------------------
167+// Cloud wiring — reads the EXECUTION_RATE_LIMITER binding from the worker env.
168+// ---------------------------------------------------------------------------
169+
170+// The DO stub's RPC surface. The binding is declared untyped in
171+// env-augment.d.ts (matching the BLOBS precedent), so the call site narrows it
172+// to the one method the class exposes.
173+type ExecutionRateLimiterStub = {
174+ readonly increment: (windowId: number) => Promise<number>;
175+};
176+
177+type RateLimiterNamespace = {
178+ readonly idFromName: (name: string) => DurableObjectId;
179+ readonly get: (id: DurableObjectId) => unknown;
180+};
181+
182+/**
183+ * Production rate limiter backed by the `EXECUTION_RATE_LIMITER` counter DO.
184+ * When the binding is absent (unit-test workers, older local setups) the
185+ * limiter is disabled: every check passes, logged once at construction.
186+ */
187+export const makeCloudExecutionRateLimiter = (): ExecutionRateLimiter => {
188+ const limit = resolveRateLimit();
189+ const namespace = (env as { EXECUTION_RATE_LIMITER?: RateLimiterNamespace })
190+ .EXECUTION_RATE_LIMITER;
191+ if (!namespace) {
192+ console.warn(
193+ "[rate-limit] EXECUTION_RATE_LIMITER binding missing; execution rate limiting disabled",
194+ );
195+ return makeExecutionRateLimiter(() => Effect.succeed(0));
196+ }
197+ return makeExecutionRateLimiter(
198+ (organizationId, windowId) =>
199+ Effect.tryPromise(() => {
200+ const stub = namespace.get(
201+ namespace.idFromName(organizationId),
202+ ) as ExecutionRateLimiterStub;
203+ return stub.increment(windowId);
204+ }),
205+ { limit },
206+ );
207+};
208+
209+/**
210+ * The per-org hourly cap: the `EXECUTION_RATE_LIMIT_PER_HOUR` env override
211+ * (parsed as a positive integer) or `EXECUTIONS_PER_ORG_PER_HOUR` when it's
212+ * unset or unparseable. The override exists so e2e can drive the backstop with
213+ * a small number of real executions; production leaves the var unset.
214+ */
215+const resolveRateLimit = (): number => {
216+ const raw = (env as { EXECUTION_RATE_LIMIT_PER_HOUR?: string }).EXECUTION_RATE_LIMIT_PER_HOUR;
217+ if (raw === undefined) return EXECUTIONS_PER_ORG_PER_HOUR;
218+ const parsed = Number.parseInt(raw, 10);
219+ return Number.isInteger(parsed) && parsed > 0 ? parsed : EXECUTIONS_PER_ORG_PER_HOUR;
220+};
apps/cloud/src/engine/execution-stack-metered.ts+30−9
@@ -28,19 +28,40 @@ import { · 27 unmodified lines
2828 import { AutumnService } from "../extensions/billing/service";
2929 import type { DbService } from "../db/db";
3030 import { CloudExecutionSeamsLayer } from "../engine/execution-stack";
31+import { makeExecutionLimitGate } from "./execution-gate";
32+import { makeCloudExecutionRateLimiter } from "./execution-rate-limit";
3133 import { withExecutionUsageTracking } from "./execution-usage";
3234
33−// Usage-metering decorator bound to the billing service. `trackExecution` is
34−// fire-and-forget (`Effect.runFork`) so the billing call can't stall a
35−// user-facing execution.
35+// Usage-metering decorator bound to the billing service, plus the two
36+// pre-execution guards this layer owns, ordered cheapest first:
37+// 1. rate-limit backstop (counter DO, independent of billing)
38+// 2. execution balance gate (Autumn check, cached 60s, fails open)
39+// 3. usage tracking — fire-and-forget (`Effect.runFork`) so the billing
40+// call can't stall a user-facing execution.
41+// The guards wrap OUTSIDE the tracker so a blocked execution is neither run
42+// nor tracked. One gate/limiter instance per layer build: the balance cache is
43+// shared by every engine this decorator produces (in the MCP session DO that's
44+// the session's engines; on the HTTP plane the boot layer builds it once).
3645 export const CloudMeteringEngineDecorator: Layer.Layer<EngineDecorator, never, AutumnService> =
3746 Layer.effect(EngineDecorator)(
38− Effect.map(AutumnService.asEffect(), (autumn): EngineDecorator["Service"] => ({
39− decorate: (engine, identity: EngineStackIdentity) =>
40− withExecutionUsageTracking(identity.organizationId, engine, (organizationId) =>
41− Effect.runFork(autumn.trackExecution(organizationId)),
42− ),
43− })),
47+ Effect.map(AutumnService.asEffect(), (autumn): EngineDecorator["Service"] => {
48+ const balanceGate = makeExecutionLimitGate((organizationId) =>
49+ autumn.checkExecutionBalance(organizationId),
50+ );
51+ const rateLimiter = makeCloudExecutionRateLimiter();
52+ return {
53+ decorate: (engine, identity: EngineStackIdentity) =>
54+ rateLimiter.decorate(
55+ identity.organizationId,
56+ balanceGate.decorate(
57+ identity.organizationId,
58+ withExecutionUsageTracking(identity.organizationId, engine, (organizationId) =>
59+ Effect.runFork(autumn.trackExecution(organizationId)),
60+ ),
61+ ),
62+ ),
63+ };
64+ }),
4465 );
4566
4667 /**
apps/cloud/src/env-augment.d.ts+14−0
@@ -32,6 +32,20 @@ declare global { · 31 unmodified lines
3232 // ON; the test workers set "true" so fixtures can reach localhost.
3333 ALLOW_LOCAL_NETWORK?: string;
3434
35+ // Per-org execution rate-limit counter DO (wrangler.jsonc
36+ // `durable_objects`). Declared optional here (matching the BLOBS
37+ // precedent) rather than regenerating worker-configuration.d.ts: test
38+ // workers and older local setups run without the binding, and the
39+ // limiter degrades to disabled when absent.
40+ EXECUTION_RATE_LIMITER?: import("@cloudflare/workers-types").DurableObjectNamespace;
41+
42+ // Optional per-org hourly execution rate-limit override, parsed as an
43+ // integer (defaults to EXECUTIONS_PER_ORG_PER_HOUR = 1000 when unset or
44+ // unparseable). Exists for e2e: the production cap of 1000/hour can't be
45+ // exercised with real executions, so the e2e dev-server env sets a small
46+ // number to drive the backstop. Production leaves it unset.
47+ EXECUTION_RATE_LIMIT_PER_HOUR?: string;
48+
3549 // Billing
3650 AUTUMN_SECRET_KEY?: string;
3751 /** Optional Autumn base-URL override (Autumn emulator in tests/dev). */
apps/cloud/src/extensions/billing/service.ts+14−1
@@ -22,6 +22,9 @@ export class AutumnError extends Data.TaggedError("AutumnError")<{ · 21 unmodified lines
2222
2323 export type IAutumnService = Readonly<{
2424 use: <A>(fn: (client: Autumn) => Promise<A>) => Effect.Effect<A, AutumnError, never>;
25+ checkExecutionBalance: (
26+ organizationId: string,
27+ ) => Effect.Effect<{ readonly allowed: boolean }, AutumnError, never>;
2528 /**
2629 * Fire-and-forget-safe execution usage tracker. Errors are caught and
2730 * logged; the returned Effect never fails. Callers typically
@@ -44,6 +47,7 @@ const make = Effect.sync(() => { · 16 unmodified lines
4447 );
4548 return {
4649 use: () => notConfigured,
50+ checkExecutionBalance: () => notConfigured,
4751 trackExecution: () => Effect.void,
4852 } satisfies IAutumnService;
4953 }
@@ -80,7 +84,16 @@ const make = Effect.sync(() => { · 30 unmodified lines
8084 );
8185 }).pipe(Effect.withSpan("autumn.trackExecution"));
8286
83− return { use, trackExecution } satisfies IAutumnService;
87+ const checkExecutionBalance = (organizationId: string) =>
88+ Effect.gen(function* () {
89+ yield* Effect.annotateCurrentSpan({ "autumn.customer.id": organizationId });
90+ const check = yield* use((c) =>
91+ c.check({ customerId: organizationId, featureId: "executions" }),
92+ );
93+ return { allowed: check.allowed };
94+ }).pipe(Effect.withSpan("autumn.checkExecutionBalance"));
95+
96+ return { use, checkExecutionBalance, trackExecution } satisfies IAutumnService;
8497 });
8598
8699 export class AutumnService extends Context.Service<AutumnService, IAutumnService>()(
apps/cloud/src/server.ts+5−0
@@ -55,6 +55,11 @@ export const McpSessionDOSqlite = Sentry.instrumentDurableObjectWithSentry( · 54 unmodified lines
5555 // (with a `deleted_classes: ["McpSessionDO"]` migration) now that nothing binds it.
5656 export class McpSessionDO extends DurableObject {}
5757
58+// Per-org execution rate-limit counter DO (abuse backstop; migration v3,
59+// `EXECUTION_RATE_LIMITER` binding). Plain counter, no Sentry wrapper needed:
60+// its callers already fail open and report errors themselves.
61+export { ExecutionRateLimiterDO } from "./engine/execution-rate-limit";
62+
5863 // ---------------------------------------------------------------------------
5964 // Worker fetch handler
6065 //
apps/cloud/wrangler.jsonc+11−0
@@ -22,6 +22,13 @@ · 21 unmodified lines
2222 "name": "MCP_SESSION",
2323 "class_name": "McpSessionDOSqlite",
2424 },
25+ // Per-org execution rate-limit counter (abuse backstop). One instance
26+ // per organization via idFromName(orgId); a single { windowId, count }
27+ // record per instance, purged by alarm after inactivity.
28+ {
29+ "name": "EXECUTION_RATE_LIMITER",
30+ "class_name": "ExecutionRateLimiterDO",
31+ },
2532 ],
2633 },
2734 // The MCP session DO moved to the Cloudflare Agents (`McpAgent`) base, which
@@ -42,6 +49,10 @@ · 14 unmodified lines
4249 "tag": "v2",
4350 "new_sqlite_classes": ["McpSessionDOSqlite"],
4451 },
52+ {
53+ "tag": "v3",
54+ "new_sqlite_classes": ["ExecutionRateLimiterDO"],
55+ },
4556 ],
4657 "services": [
4758 {
bun.lock+2−2
@@ -344,7 +344,7 @@ · 343 unmodified lines
344344 "version": "0.0.24",
345345 "dependencies": {
346346 "@executor-js/api": "workspace:*",
347− "@executor-js/emulate": "^0.10.0",
347+ "@executor-js/emulate": "^0.13.2",
348348 "@executor-js/mcporter": "^0.11.4",
349349 "@executor-js/plugin-graphql": "workspace:*",
350350 "@executor-js/plugin-mcp": "workspace:*",
@@ -1743,7 +1743,7 @@ · 1392 unmodified lines
17431743
17441744 "@executor-js/e2e": ["@executor-js/e2e@workspace:e2e"],
17451745
1746− "@executor-js/emulate": ["@executor-js/emulate@0.10.0", "", { "dependencies": { "@aws-sdk/client-s3": "^3.1031.0", "@aws-sdk/client-sqs": "^3.1075.0", "@azure/msal-node": "^5.3.0", "@clerk/backend": "^3.8.4", "@octokit/rest": "^22.0.1", "@okta/okta-auth-js": "^8.0.1", "@slack/web-api": "^7.16.0", "@vercel/sdk": "^1.28.4", "@workos-inc/node": "^8.13.0", "atlas-api-client": "^0.3.0", "autumn-js": "^1.2.8", "commander": "^14", "googleapis": "^173.0.0", "graphql": "^16.9.0", "graphql-request": "^7.4.0", "openid-client": "^6.8.4", "picocolors": "^1.1.1", "resend": "^6.16.0", "spotify-web-api-node": "^5.0.2", "stripe": "^22.3.0", "twitter-api-v2": "^1.29.0", "yaml": "^2" }, "bin": { "emulate": "dist/index.js" } }, "sha512-GE1+XDQ4FJt4ZDrwNjuUqUEG1WaH06UE12ME/xJdcCNbsa6EE6SA+i8onVJQ5Dr7DbUveU37E4djLTiAcceLPw=="],
1746+ "@executor-js/emulate": ["@executor-js/emulate@0.13.2", "", { "dependencies": { "@aws-sdk/client-s3": "^3.1031.0", "@aws-sdk/client-sqs": "^3.1075.0", "@azure/msal-node": "^5.3.0", "@clerk/backend": "^3.8.4", "@octokit/rest": "^22.0.1", "@okta/okta-auth-js": "^8.0.1", "@slack/web-api": "^7.16.0", "@vercel/sdk": "^1.28.4", "@workos-inc/node": "^8.13.0", "atlas-api-client": "^0.3.0", "autumn-js": "^1.2.8", "commander": "^14", "googleapis": "^173.0.0", "graphql": "^16.9.0", "graphql-request": "^7.4.0", "openid-client": "^6.8.4", "picocolors": "^1.1.1", "resend": "^6.16.0", "spotify-web-api-node": "^5.0.2", "stripe": "^22.3.0", "twitter-api-v2": "^1.29.0", "yaml": "^2" }, "bin": { "emulate": "dist/index.js" } }, "sha512-ikmzZgBrRnmJgKKOW9TE2KCCdenKtihx2nWW/mMMmugn3UlVJsxuIxp6jv0BEo4GZcY8Ve2oka6UEKl+gBKRZg=="],
17471747
17481748 "@executor-js/example-all-plugins": ["@executor-js/example-all-plugins@workspace:examples/all-plugins"],
17491749
e2e/cloud/mcp-execution-limits.test.ts+200−0
@@ -0,0 +1,200 @@
1+// Cloud-only (billing): the pre-execution guards that sit in front of the MCP
2+// session Durable Object's execution engine. Two independent gates run before
3+// every new `execute`:
4+//
5+// - the BALANCE gate consults Autumn's `balances.check` and blocks once an
6+// org has used its plan's included executions, and
7+// - the RATE-LIMIT backstop counts executions per org per hour and blocks
8+// runaway automation even when billing is down.
9+//
10+// Both FAIL OPEN: an Autumn error or timeout must never take executions down
11+// with the billing provider. These scenarios pin all four behaviours end to
12+// end against the real workerd + McpSessionDO topology, driving a real MCP
13+// client and reading Autumn's own ledger as ground truth — a blocked execution
14+// must not reach the meter, and a failed-open one must show the check was
15+// actually attempted (and faulted) rather than silently skipped.
16+import { expect } from "@effect/vitest";
17+import { Effect } from "effect";
18+
19+import {
20+ EXECUTION_LIMIT_BLOCKED_MESSAGE,
21+ RATE_LIMIT_BLOCKED_MESSAGE,
22+} from "../../apps/cloud/src/engine/execution-limit-messages";
23+import { scenario } from "../src/scenario";
24+import { Autumn, Billing, Mcp, Target } from "../src/services";
25+import { E2E_EXECUTION_RATE_LIMIT } from "../setup/execution-limits";
26+import type { Identity } from "../src/target";
27+
28+const emailOf = (identity: Identity): string => identity.credentials?.email ?? identity.label;
29+
30+/** The org the bearer is scoped to — the Autumn customer id the guards decide
31+ * against — read from the JWT's public claims. */
32+const orgIdOf = (bearer: string): string => {
33+ const claims = JSON.parse(Buffer.from(bearer.split(".")[1] ?? "", "base64url").toString()) as {
34+ readonly org_id?: string;
35+ };
36+ if (!claims.org_id) throw new Error("orgIdOf: bearer carries no org_id claim");
37+ return claims.org_id;
38+};
39+
40+// The e2e worker is booted with a small EXECUTION_RATE_LIMIT_PER_HOUR so the
41+// backstop is reachable with real executions (cloud.boot.ts). The prod cap of
42+// 1000/hour can't be. One shared constant (setup/execution-limits.ts) keeps
43+// this and the boot env from drifting apart.
44+const RATE_LIMIT = E2E_EXECUTION_RATE_LIMIT;
45+
46+scenario(
47+ "Billing · an org out of executions is blocked before the code ever runs",
48+ { timeout: 180_000 },
49+ Effect.gen(function* () {
50+ // Gates: billing is enforced here AND the Autumn ledger is observable.
51+ // Yield before any work so a target missing either capability skips cleanly.
52+ yield* Billing;
53+ const autumn = yield* Autumn;
54+ const target = yield* Target;
55+ const mcp = yield* Mcp;
56+
57+ const identity = yield* target.newIdentity();
58+ const bearer = yield* mcp.mintBearer(emailOf(identity));
59+ const customerId = orgIdOf(bearer);
60+
61+ // Burn the whole included allotment so the org's remaining balance is zero.
62+ // This records one `balances.track` event (the plan's included amount) —
63+ // the baseline the blocked execution is measured against below.
64+ yield* autumn.exhaustExecutions(customerId);
65+ const afterExhaust = yield* autumn.usageEvents({ customerId, featureId: "executions" });
66+ expect(afterExhaust.length, "exhausting the balance records exactly one usage event").toBe(1);
67+
68+ // A NEW MCP session => a fresh session DO => a fresh gate cache, so the
69+ // balance is checked live (no 60s stale-allow window from a prior session).
70+ const session = mcp.session(identity);
71+ const result = yield* session.call("execute", { code: "return 1 + 1;" });
72+
73+ expect(result.ok, "a blocked execution surfaces as an MCP error result").toBe(false);
74+ expect(result.text, "the client sees the execution-limit message").toContain(
75+ EXECUTION_LIMIT_BLOCKED_MESSAGE,
76+ );
77+
78+ // The meter is the source of truth: a blocked execution is neither run nor
79+ // usage-tracked. Give any (erroneous) fire-and-forget track ample time to
80+ // land, then assert nothing beyond the exhaust event was recorded.
81+ const settle = yield* autumn.usageEvents({ customerId, featureId: "executions" });
82+ expect(settle.length, "the blocked execution adds no 'executions' usage event").toBe(
83+ afterExhaust.length,
84+ );
85+ }),
86+);
87+
88+scenario(
89+ "Billing · a balance-check failure fails open and still runs the execution",
90+ { timeout: 180_000 },
91+ Effect.gen(function* () {
92+ yield* Billing;
93+ const autumn = yield* Autumn;
94+ const target = yield* Target;
95+ const mcp = yield* Mcp;
96+
97+ const identity = yield* target.newIdentity();
98+
99+ yield* Effect.gen(function* () {
100+ // One-shot 500 on the very next balances.check the gate makes.
101+ yield* autumn.armFault({
102+ match: { operationId: "balances.check" },
103+ response: { status: 500, body: { message: "emulated billing outage" } },
104+ times: 1,
105+ });
106+
107+ const session = mcp.session(identity);
108+ const result = yield* session.call("execute", { code: "return 6 * 7;" });
109+
110+ // Fail open: the outage must not block a solvent org's execution.
111+ expect(result.ok, "the execution succeeds despite the billing outage").toBe(true);
112+ expect(result.text, "it returns its value").toContain("42");
113+
114+ // Proof the gate actually CONSULTED Autumn and failed open (rather than
115+ // never checking): the faulted balances.check is in the emulator ledger.
116+ const checks = yield* autumn.ledgerFor("balances.check");
117+ expect(
118+ checks.some((entry) => entry.faulted),
119+ "the gate's balances.check reached Autumn and was faulted",
120+ ).toBe(true);
121+ }).pipe(Effect.ensuring(autumn.clearFaults().pipe(Effect.ignore)));
122+ }),
123+);
124+
125+scenario(
126+ "Billing · a balance-check timeout fails open and still runs the execution",
127+ { timeout: 180_000 },
128+ Effect.gen(function* () {
129+ yield* Billing;
130+ const autumn = yield* Autumn;
131+ const target = yield* Target;
132+ const mcp = yield* Mcp;
133+
134+ const identity = yield* target.newIdentity();
135+
136+ yield* Effect.gen(function* () {
137+ // The armed check stalls 3s before returning — past the gate's 2s budget,
138+ // so the gate times out client-side and fails open. (The response still
139+ // arrives after the delay; the ~2s added latency fits the timeout.)
140+ yield* autumn.armFault({
141+ match: { operationId: "balances.check" },
142+ response: { status: 200, body: { allowed: true } },
143+ times: 1,
144+ delayMs: 3000,
145+ });
146+
147+ const session = mcp.session(identity);
148+ const result = yield* session.call("execute", { code: "return 21 * 2;" });
149+
150+ expect(result.ok, "the execution succeeds despite the slow balance check").toBe(true);
151+ expect(result.text, "it returns its value").toContain("42");
152+
153+ const checks = yield* autumn.ledgerFor("balances.check");
154+ expect(checks.length, "the gate did attempt the (stalled) balances.check").toBeGreaterThan(0);
155+ }).pipe(Effect.ensuring(autumn.clearFaults().pipe(Effect.ignore)));
156+ }),
157+);
158+
159+scenario(
160+ "Billing · the rate-limit backstop blocks runaway executions and doesn't meter them",
161+ { timeout: 180_000 },
162+ Effect.gen(function* () {
163+ yield* Billing;
164+ const autumn = yield* Autumn;
165+ const target = yield* Target;
166+ const mcp = yield* Mcp;
167+
168+ const identity = yield* target.newIdentity();
169+ const bearer = yield* mcp.mintBearer(emailOf(identity));
170+ const customerId = orgIdOf(bearer);
171+
172+ // All executions run in ONE session so the balance cache stays allowed
173+ // (the org is solvent) — only the per-org hourly counter should trip.
174+ const session = mcp.session(identity);
175+
176+ // Run exactly the limit's worth of allowed executions.
177+ for (let i = 1; i <= RATE_LIMIT; i++) {
178+ const ok = yield* session.call("execute", { code: `return ${i};` });
179+ expect(ok.ok, `execution ${i} (within the limit) succeeds`).toBe(true);
180+ }
181+
182+ // The next one crosses the hourly cap: blocked with the backstop message.
183+ const blocked = yield* session.call("execute", { code: "return 999;" });
184+ expect(blocked.ok, "the execution past the cap is an MCP error result").toBe(false);
185+ expect(blocked.text, "the client sees the rate-limit message").toContain(
186+ RATE_LIMIT_BLOCKED_MESSAGE,
187+ );
188+
189+ // A rate-limited execution is neither run nor metered: exactly the allowed
190+ // runs reached the Autumn meter, and the blocked one added nothing.
191+ const metered = yield* autumn.expectUsage({
192+ customerId,
193+ featureId: "executions",
194+ count: RATE_LIMIT,
195+ });
196+ expect(metered.length, "only the allowed executions are metered — the blocked one is not").toBe(
197+ RATE_LIMIT,
198+ );
199+ }),
200+);
e2e/package.json+1−1
@@ -22,7 +22,7 @@ · 21 unmodified lines
2222 },
2323 "dependencies": {
2424 "@executor-js/api": "workspace:*",
25− "@executor-js/emulate": "^0.10.0",
25+ "@executor-js/emulate": "^0.13.2",
2626 "@executor-js/mcporter": "^0.11.4",
2727 "@executor-js/plugin-graphql": "workspace:*",
2828 "@executor-js/plugin-mcp": "workspace:*",
e2e/setup/autumn-plans.ts+18−0
@@ -49,3 +49,21 @@ const toSeed = (plan: AtmnPlan): AutumnSeedPlan => ({ · 48 unmodified lines
4949 export const AUTUMN_PLAN_SEED: AutumnSeedPlan[] = [free, team, enterprise].map((p) =>
5050 toSeed(p as AtmnPlan),
5151 );
52+
53+/**
54+ * The "executions" allotment a fresh org has on the auto-enabled default plan
55+ * (Free): the exact number of executions it takes to exhaust the quota. Read
56+ * from the same seed the emulator is booted with, so a plan change in
57+ * autumn.config.ts flows straight into the balance-gate scenarios rather than
58+ * drifting against a hardcoded magic number.
59+ */
60+export const DEFAULT_PLAN_EXECUTIONS_INCLUDED: number = (() => {
61+ const defaultPlan = AUTUMN_PLAN_SEED.find((plan) => plan.auto_enable);
62+ const executions = defaultPlan?.items.find((item) => item.feature_id === "executions");
63+ if (executions?.included === undefined) {
64+ throw new Error(
65+ "autumn-plans: the auto-enabled default plan has no included 'executions' amount to exhaust",
66+ );
67+ }
68+ return executions.included;
69+})();
e2e/setup/cloud.boot.ts+7−0
@@ -12,6 +12,7 @@ import { createEmulator } from "@executor-js/emulate"; · 11 unmodified lines
1212
1313 import { bootProcesses, waitForHttp } from "./boot";
1414 import { AUTUMN_PLAN_SEED } from "./autumn-plans";
15+import { E2E_EXECUTION_RATE_LIMIT } from "./execution-limits";
1516
1617 export const cloudDir = fileURLToPath(new URL("../../apps/cloud/", import.meta.url));
1718
@@ -90,6 +91,12 @@ export const bootCloud = async (options: CloudBootOptions): Promise<CloudBooted> · 72 unmodified lines
9091 MCP_AUTHKIT_DOMAIN: workosUrl,
9192 MCP_RESOURCE_ORIGIN: options.publicUrl,
9293 ALLOW_LOCAL_NETWORK: "true",
94+ // Shrink the per-org hourly execution cap (prod default 1000) to a number
95+ // the rate-limit-backstop scenario can actually exhaust with real
96+ // executions — but see execution-limits.ts: it must stay above every other
97+ // scenario's per-org execute count. Reaches the worker via
98+ // CLOUDFLARE_INCLUDE_PROCESS_ENV, same as ALLOW_LOCAL_NETWORK.
99+ EXECUTION_RATE_LIMIT_PER_HOUR: String(E2E_EXECUTION_RATE_LIMIT),
93100 // Throwaway PGlite on its own port + dir so it never fights `bun dev`.
94101 DEV_DB_PORT: String(options.dbPort),
95102 DEV_DB_PATH: dbPath,
e2e/setup/execution-limits.ts+14−0
@@ -0,0 +1,14 @@
1+// The e2e worker's per-org hourly execution cap (EXECUTION_RATE_LIMIT_PER_HOUR).
2+// One constant with two consumers, the boot recipe env (cloud.boot.ts) and the
3+// rate-limit backstop scenario (cloud/mcp-execution-limits.test.ts), so they
4+// cannot drift apart.
5+//
6+// Picking the value is a squeeze from both sides. It must be LOW enough that
7+// the backstop scenario can exhaust it with real sequential executions (prod's
8+// 1000/hour cannot be reached in a test), and HIGH enough that no other
9+// scenario trips it: the counter is per organization and every `execute` a
10+// scenario runs counts against its org, so this must exceed the busiest
11+// single-org scenario's execute count (currently toolkits-mcp at ~8) with
12+// comfortable headroom. If a scenario ever fails with the rate-limit backstop
13+// message, it outgrew this cap: raise it here, never in the boot env alone.
14+export const E2E_EXECUTION_RATE_LIMIT = 20;
e2e/src/surfaces/autumn.ts+123−0
@@ -9,6 +9,8 @@ · 8 unmodified lines
99 // so the contract has to be pinned where the usage is read.
1010 import { Effect, Schedule } from "effect";
1111
12+import { DEFAULT_PLAN_EXECUTIONS_INCLUDED } from "../../setup/autumn-plans";
13+
1214 /** One usage event Autumn recorded: a `track` of `value` units of `featureId`
1315 * for `customerId` (the organization the execution ran under). */
1416 export interface UsageEvent {
@@ -24,6 +26,33 @@ export interface UsageQuery { · 9 unmodified lines
2426 readonly featureId: string;
2527 }
2628
29+/** A one-shot (or counted) fault to arm against a matching Autumn request. Mirrors
30+ * the emulator's `POST /_emulate/faults` body: a request matching every provided
31+ * `match` criterion short-circuits with `response`, `times` decrements per hit
32+ * (default 1, removed at zero), and `delayMs` stalls before the fault returns —
33+ * the knob a timeout scenario uses to push Autumn past the gate's time budget. */
34+export interface FaultInput {
35+ readonly match: {
36+ readonly operationId?: string;
37+ readonly method?: string;
38+ readonly pathPattern?: string;
39+ };
40+ readonly response: { readonly status: number; readonly body?: unknown };
41+ readonly times?: number;
42+ readonly delayMs?: number;
43+}
44+
45+/** One request the emulator recorded in its ledger, narrowed to what these
46+ * scenarios assert on: which operation ran and whether a fault short-circuited
47+ * it. A faulted `balances.check` entry is the proof the gate actually consulted
48+ * Autumn (and then failed open) rather than never checking at all. */
49+export interface LedgerEntry {
50+ readonly operationId?: string;
51+ readonly method: string;
52+ readonly path: string;
53+ readonly faulted: boolean;
54+}
55+
2756 export interface AutumnSurface {
2857 /** One-shot read of the matching usage events from the ledger. */
2958 readonly usageEvents: (query: UsageQuery) => Effect.Effect<readonly UsageEvent[], unknown>;
@@ -41,6 +70,20 @@ export interface AutumnSurface { · 11 unmodified lines
4170 * the billing backend becomes consistent. The `sessionId` is the last path
4271 * segment of the hosted checkout URL the browser was sent to. */
4372 readonly settleCheckout: (sessionId: string) => Effect.Effect<void, unknown>;
73+ /** Burn an org's entire remaining "executions" balance in one `balances.track`,
74+ * so the next `balances.check` reports `allowed: false`. The amount is the
75+ * default plan's included allotment (read from the plan seed, never hardcoded),
76+ * which drives `remaining` to zero. This is the setup a "blocked at cap"
77+ * scenario runs before opening the session it expects to be gated. */
78+ readonly exhaustExecutions: (customerId: string) => Effect.Effect<void, unknown>;
79+ /** Arm a fault against a matching Autumn request (`POST /_emulate/faults`). */
80+ readonly armFault: (input: FaultInput) => Effect.Effect<void, unknown>;
81+ /** Clear every armed fault (`DELETE /_emulate/faults`) — the finalizer that
82+ * keeps a scenario's fault from leaking into the next one. */
83+ readonly clearFaults: () => Effect.Effect<void, unknown>;
84+ /** Read the emulator's request ledger, filtered to one operation — used to
85+ * prove a faulted `balances.check` was actually attempted (fail-open proof). */
86+ readonly ledgerFor: (operationId: string) => Effect.Effect<readonly LedgerEntry[], unknown>;
4487 }
4588
4689 export const makeAutumnSurface = (autumnUrl: string): AutumnSurface => {
@@ -90,9 +133,89 @@ export const makeAutumnSurface = (autumnUrl: string): AutumnSurface => { · 43 unmodified lines
90133 }
91134 });
92135
136+ // Track exactly the plan's included allotment in one event, driving
137+ // `remaining` (included - usage) to zero so the next check reports blocked.
138+ const exhaustExecutions = (customerId: string) =>
139+ Effect.gen(function* () {
140+ const response = yield* Effect.promise(() =>
141+ fetch(`${autumnUrl}/v1/balances.track`, {
142+ method: "POST",
143+ headers: { "content-type": "application/json" },
144+ body: JSON.stringify({
145+ customer_id: customerId,
146+ feature_id: "executions",
147+ value: DEFAULT_PLAN_EXECUTIONS_INCLUDED,
148+ }),
149+ }),
150+ );
151+ if (!response.ok) {
152+ return yield* Effect.fail(
153+ `autumn balances.track responded ${response.status}: ${yield* Effect.promise(() => response.text())}`,
154+ );
155+ }
156+ });
157+
158+ const armFault = (input: FaultInput) =>
159+ Effect.gen(function* () {
160+ const response = yield* Effect.promise(() =>
161+ fetch(`${autumnUrl}/_emulate/faults`, {
162+ method: "POST",
163+ headers: { "content-type": "application/json" },
164+ body: JSON.stringify(input),
165+ }),
166+ );
167+ if (!response.ok) {
168+ return yield* Effect.fail(
169+ `autumn arm fault responded ${response.status}: ${yield* Effect.promise(() => response.text())}`,
170+ );
171+ }
172+ });
173+
174+ const clearFaults = () =>
175+ Effect.gen(function* () {
176+ const response = yield* Effect.promise(() =>
177+ fetch(`${autumnUrl}/_emulate/faults`, { method: "DELETE" }),
178+ );
179+ if (!response.ok) {
180+ return yield* Effect.fail(
181+ `autumn clear faults responded ${response.status}: ${yield* Effect.promise(() => response.text())}`,
182+ );
183+ }
184+ });
185+
186+ const ledgerFor = (operationId: string) =>
187+ Effect.gen(function* () {
188+ const response = yield* Effect.promise(() => fetch(`${autumnUrl}/_emulate/ledger`));
189+ if (!response.ok) {
190+ return yield* Effect.fail(
191+ `autumn ledger responded ${response.status}: ${yield* Effect.promise(() => response.text())}`,
192+ );
193+ }
194+ const body = (yield* Effect.promise(() => response.json())) as {
195+ readonly entries?: ReadonlyArray<{
196+ readonly operationId?: string;
197+ readonly method?: string;
198+ readonly path?: string;
199+ readonly faulted?: boolean;
200+ }>;
201+ };
202+ return (body.entries ?? [])
203+ .filter((entry) => entry.operationId === operationId)
204+ .map((entry) => ({
205+ operationId: entry.operationId,
206+ method: entry.method ?? "",
207+ path: entry.path ?? "",
208+ faulted: entry.faulted === true,
209+ }));
210+ });
211+
93212 return {
94213 usageEvents,
95214 settleCheckout,
215+ exhaustExecutions,
216+ armFault,
217+ clearFaults,
218+ ledgerFor,
96219 expectUsage: (query) =>
97220 usageEvents(query).pipe(
98221 Effect.filterOrFail(