History

Enforce execution limits with a balance gate and per-org rate-limit backstop (#1274)

* Add execution balance check to billing service * Add pre-execution balance gate for org execution limits Check the org's Autumn execution balance before execute/executeWithPause (never resume, so paused executions can always complete). Blocked orgs get a descriptive ExecuteResult.error instead of running. Fails open on any billing error or a 2s timeout, and caches per-org outcomes for 60s. * Add per-org execution rate-limit backstop 1000 execute calls per org per hour, fixed window, counted in a minimal per-org Durable Object (EXECUTION_RATE_LIMITER, one instance per org via idFromName). Independent of billing so runaway automation is caught even when Autumn is down; fails open if the counter DO errors or is slow. The DO stores a single {windowId, count} record and purges itself by alarm after two idle windows. * Wire execution guards into the metered engine decorator Order: rate-limit backstop first (cheap counter), then the balance gate, then usage tracking. Guards wrap outside the tracker so a blocked execution is neither run nor tracked. Covers both planes (HTTP executor plane and MCP session DO) since both build engines through this layer. * Test the execution balance gate and rate-limit backstop Covers allow/block, the typed errors, fail-open on billing errors and timeouts (asserting the check was attempted AND the execution ran), the 60s per-org outcome cache (allowed and blocked cached, errors never), window rollover, per-org isolation, and that resume is never gated. * Use typed test errors and instanceof assertions in gate tests * Replace execution-limit unit tests with e2e coverage Delete the balance-gate and rate-limit unit tests: the guards are now exercised end to end against the real workerd + Autumn emulator, which is a truer test of behaviour that mattered here (blocked executions not reaching the meter, failing open on billing errors). Drop the test-only affordances the unit tests were the only callers of: the gate's `timeoutMs` option knob and both limiters' `check` diagnostic methods. Move the two blocked-execution messages into a small Cloudflare-free module so the e2e suite can import them as assertion ground truth without pulling the guards' worker-runtime dependencies into its typecheck. * Make the per-org execution rate limit env-configurable Add an optional EXECUTION_RATE_LIMIT_PER_HOUR override, parsed as a positive integer and falling back to the 1000/hour default when unset or unparseable. Production leaves it unset; it exists so e2e can drive the abuse backstop with a handful of real executions instead of a thousand. The e2e cloud dev server sets it to 3, reaching the worker the same way ALLOW_LOCAL_NETWORK does. * Add Autumn e2e helpers for fault injection and balance exhaustion Bump the emulator to 0.13.2 for its balances.check support and one-shot fault injection, and extend the Autumn surface to use them: armFault/clearFaults drive the emulator's fault control plane, exhaustExecutions burns an org's whole included allotment in one track (the amount read from the plan seed, never hardcoded), and ledgerFor reads the request ledger so a scenario can prove a faulted balances.check was actually attempted. * Add e2e scenarios for the pre-execution limit guards Four cloud scenarios drive a real MCP client against the workerd + session-DO topology and read Autumn's ledger as ground truth: an org at its cap is blocked before the code runs and not metered; a balance-check 500 and a balance-check timeout each fail open and still run the execution (with the faulted check visible in the ledger as proof the gate consulted Autumn); and the rate-limit backstop blocks the execution past the hourly cap without metering it. Faults are one-shot and cleared in finalizers, and each scenario uses a fresh org. * Raise the e2e execution rate cap above other scenarios' usage The e2e worker's per-org hourly cap of 3 was low enough that ordinary scenarios (toolkits-mcp, no-auth-connection, connect-handoff) tripped the backstop mid-test once the suite ran them against the shared worker. Lift it to 20 (still exhaustible by the backstop scenario) and move the value into one shared constant consumed by both the boot env and the test.

Rhys Sullivan committed 4 months ago

Commit cfd02cf1 parent 718ffd215 files · +855 −13Browse files

Unified · Split

15 files
  1. apps/cloud/src/engine/execution-gate.ts +185 −0
  2. apps/cloud/src/engine/execution-limit-messages.ts +11 −0
  3. apps/cloud/src/engine/execution-rate-limit.ts +220 −0
  4. apps/cloud/src/engine/execution-stack-metered.ts +30 −9
  5. apps/cloud/src/env-augment.d.ts +14 −0
  6. apps/cloud/src/extensions/billing/service.ts +14 −1
  7. apps/cloud/src/server.ts +5 −0
  8. apps/cloud/wrangler.jsonc +11 −0
  9. bun.lock +2 −2
  10. e2e/cloud/mcp-execution-limits.test.ts +200 −0
  11. e2e/package.json +1 −1
  12. e2e/setup/autumn-plans.ts +18 −0
  13. e2e/setup/cloud.boot.ts +7 −0
  14. e2e/setup/execution-limits.ts +14 −0
  15. e2e/src/surfaces/autumn.ts +123 −0
apps/cloud/src/engine/execution-gate.ts+185−0
@@ -0,0 +1,185 @@
1// ---------------------------------------------------------------------------
2// Pre-execution balance gate — blocks new executions once an organization has
3// used up its plan's included executions.
4//
5// Usage is tracked to Autumn after every execution (execution-usage.ts), but
6// nothing ever CHECKED the balance before running, so a free-tier org could run
7// unbounded executions past its quota. This gate consults
8// `AutumnService.checkExecutionBalance` before `execute` / `executeWithPause`.
9// `resume` is never gated: a paused execution already consumed its quota slot
10// when it started, and blocking resume would strand approved work forever.
11//
12// FAIL OPEN is a hard requirement: any Autumn error, timeout, or missing
13// customer/feature allows the execution (logged + Sentry, mirroring
14// `trackExecution`'s reporting). Autumn being down must never block
15// executions or add meaningful latency, so the check is bounded by a short
16// timeout and its outcome is cached per organization.
17//
18// Error surfacing: the gate fails internally with the typed
19// `ExecutionLimitReachedError`, and the engine seam folds it into the
20// descriptive `ExecuteResult.error` channel. That is the codebase's mechanism
21// for getting a domain failure's message to the MCP client verbatim — engine
22// error-channel failures are deliberately rendered opaque by the MCP host
23// ("Internal tool error [correlation id]"), exactly like the runtimes fold
24// `CodeCompilationError` / `SandboxRuntimeError` into `ExecuteResult.error`.
25// ---------------------------------------------------------------------------
26
27import * as Sentry from "@sentry/cloudflare";
28import { Data, Effect } from "effect";
29import type * as Cause from "effect/Cause";
30
31import type { ExecutionEngine, ExecutionResult } from "@executor-js/execution";
32
33import { EXECUTION_LIMIT_BLOCKED_MESSAGE } from "./execution-limit-messages";
34
35// The engine's completed-result payload (`ExecuteResult` in codemode-core),
36// derived from the public execution types so this app package doesn't need a
37// direct dependency on the kernel package that declares it.
38type EngineExecuteResult = Extract<ExecutionResult, { status: "completed" }>["result"];
39
40// One check per org per minute is fresh enough for a monthly quota; both
41// allowed AND blocked outcomes are cached (errors never are).
42const BALANCE_CACHE_TTL_MS = 60_000;
43// Autumn slower than this => fail open rather than stall a user-facing
44// execution behind the billing provider.
45const BALANCE_CHECK_TIMEOUT_MS = 2_000;
46// Sweep guard so a long-lived worker isolate serving many orgs can't grow the
47// cache map unbounded.
48const BALANCE_CACHE_MAX_ENTRIES = 10_000;
49
50export { EXECUTION_LIMIT_BLOCKED_MESSAGE };
51
52export class ExecutionLimitReachedError extends Data.TaggedError("ExecutionLimitReachedError")<{
53 readonly organizationId: string;
54 readonly message: string;
55}> {}
56
57/** Internal sentinel for a balance check that exceeded its time budget. */
58class GateCheckTimeoutError extends Data.TaggedError("GateCheckTimeoutError")<{
59 readonly timeoutMs: number;
60}> {}
61
62// ---------------------------------------------------------------------------
63// Shared gate seam — used by this gate and the rate-limit backstop.
64// ---------------------------------------------------------------------------
65
66export type GateDecision =
67 | { readonly blocked: false }
68 | { readonly blocked: true; readonly error: { readonly message: string } };
69
70/**
71 * Wrap an engine so `decide` runs before `execute` / `executeWithPause`. A
72 * blocked decision short-circuits to a descriptive `ExecuteResult.error`
73 * (which `formatExecuteResult` renders as a clean `isError` MCP tool result)
74 * WITHOUT invoking the inner engine — so a blocked execution is neither run
75 * nor usage-tracked. `resume` and all read-only members pass through
76 * untouched: paused executions must always be able to complete.
77 */
78export const withPreExecutionGate = <E extends Cause.YieldableError>(
79 engine: ExecutionEngine<E>,
80 decide: Effect.Effect<GateDecision>,
81): ExecutionEngine<E> => ({
82 execute: (code, options) =>
83 Effect.flatMap(
84 decide,
85 (decision): Effect.Effect<EngineExecuteResult, E> =>
86 decision.blocked
87 ? Effect.succeed({ result: null, error: decision.error.message })
88 : engine.execute(code, options),
89 ),
90 executeWithPause: (code, options) =>
91 Effect.flatMap(
92 decide,
93 (decision): Effect.Effect<ExecutionResult, E> =>
94 decision.blocked
95 ? Effect.succeed({
96 status: "completed",
97 result: { result: null, error: decision.error.message },
98 })
99 : engine.executeWithPause(code, options),
100 ),
101 // resume is never gated: paused executions must be able to complete.
102 resume: (executionId, response) => engine.resume(executionId, response),
103 getPausedExecution: (executionId) => engine.getPausedExecution(executionId),
104 pausedExecutionCount: () => engine.pausedExecutionCount(),
105 hasPausedExecutions: () => engine.hasPausedExecutions(),
106 getDescription: engine.getDescription,
107});
108
109// ---------------------------------------------------------------------------
110// Balance gate factory
111// ---------------------------------------------------------------------------
112
113export type ExecutionBalanceCheck = (
114 organizationId: string,
115) => Effect.Effect<{ readonly allowed: boolean }, unknown>;
116
117/**
118 * Build a balance gate around `checkBalance` (in production:
119 * `AutumnService.checkExecutionBalance`). One gate instance holds one
120 * per-organization outcome cache; the metered decorator layer creates a single
121 * instance so all engines it decorates share it.
122 */
123export const makeExecutionLimitGate = (checkBalance: ExecutionBalanceCheck) => {
124 const timeoutMs = BALANCE_CHECK_TIMEOUT_MS;
125 const cache = new Map<string, { readonly allowed: boolean; readonly expiresAtMs: number }>();
126
127 const writeCache = (organizationId: string, allowed: boolean, nowMs: number): void => {
128 if (cache.size >= BALANCE_CACHE_MAX_ENTRIES) {
129 for (const [key, entry] of cache) {
130 if (entry.expiresAtMs <= nowMs) cache.delete(key);
131 }
132 // Still saturated after dropping expired entries: reset rather than grow.
133 if (cache.size >= BALANCE_CACHE_MAX_ENTRIES) cache.clear();
134 }
135 cache.set(organizationId, { allowed, expiresAtMs: nowMs + BALANCE_CACHE_TTL_MS });
136 };
137
138 const toDecision = (organizationId: string, allowed: boolean): GateDecision =>
139 allowed
140 ? { blocked: false }
141 : {
142 blocked: true,
143 error: new ExecutionLimitReachedError({
144 organizationId,
145 message: EXECUTION_LIMIT_BLOCKED_MESSAGE,
146 }),
147 };
148
149 const decide = (organizationId: string): Effect.Effect<GateDecision> =>
150 Effect.suspend(() => {
151 const nowMs = Date.now();
152 const cached = cache.get(organizationId);
153 if (cached && cached.expiresAtMs > nowMs) {
154 return Effect.succeed(toDecision(organizationId, cached.allowed));
155 }
156 return checkBalance(organizationId).pipe(
157 Effect.timeoutOrElse({
158 duration: `${timeoutMs} millis`,
159 orElse: () => Effect.fail(new GateCheckTimeoutError({ timeoutMs })),
160 }),
161 Effect.map(({ allowed }) => {
162 writeCache(organizationId, allowed, nowMs);
163 return toDecision(organizationId, allowed);
164 }),
165 // FAIL OPEN: Autumn errors, timeouts, and missing customers/features
166 // must never block executions. Reported like `trackExecution` so a
167 // billing outage still pages; the error outcome is never cached.
168 Effect.catch((error: unknown) =>
169 Effect.sync((): GateDecision => {
170 console.warn("[billing] execution balance check failed open:", error);
171 Sentry.captureException(error);
172 return { blocked: false };
173 }),
174 ),
175 );
176 });
177
178 return {
179 /** Wrap an engine so the balance gate runs before each new execution. */
180 decorate: <E extends Cause.YieldableError>(
181 organizationId: string,
182 engine: ExecutionEngine<E>,
183 ): ExecutionEngine<E> => withPreExecutionGate(engine, decide(organizationId)),
184 };
185};
apps/cloud/src/engine/execution-limit-messages.ts+11−0
@@ -0,0 +1,11 @@
1// The user-facing messages a blocked execution surfaces to the MCP client, kept
2// in their own Cloudflare-free module so the e2e suite can import them as
3// ground truth without pulling the guards' `cloudflare:workers` dependencies
4// into its (worker-types-free) typecheck. The guard modules re-export these, so
5// production call sites are unchanged.
6
7export const EXECUTION_LIMIT_BLOCKED_MESSAGE =
8 "Execution limit reached: your plan's included executions for this billing period are used up. Upgrade your plan or wait for the reset to continue.";
9
10export const RATE_LIMIT_BLOCKED_MESSAGE =
11 "Rate limit exceeded: too many executions this hour. This is an abuse backstop — contact support if you hit this legitimately.";
apps/cloud/src/engine/execution-rate-limit.ts+220−0
@@ -0,0 +1,220 @@
1// ---------------------------------------------------------------------------
2// Per-org execution rate limit — an abuse backstop independent of billing.
3//
4// The balance gate (execution-gate.ts) depends on Autumn and fails open, so a
5// billing outage plus runaway automation could still run unbounded executions.
6// This limiter counts `execute` calls per organization in a fixed hourly
7// window, backed by a minimal counter Durable Object (cross-session state:
8// each MCP session lives in its own DO instance, so an in-memory counter
9// would be per-session and trivially bypassed by opening more sessions).
10//
11// Like the balance gate it FAILS OPEN: an unreachable counter DO, a missing
12// binding, or a slow call allows the execution (warn + Sentry). The backstop
13// must never take executions down with it.
14// ---------------------------------------------------------------------------
15
16import { DurableObject, env } from "cloudflare:workers";
17import * as Sentry from "@sentry/cloudflare";
18import { Data, Effect } from "effect";
19import type * as Cause from "effect/Cause";
20
21import type { ExecutionEngine } from "@executor-js/execution";
22
23import { withPreExecutionGate, type GateDecision } from "./execution-gate";
24import { RATE_LIMIT_BLOCKED_MESSAGE } from "./execution-limit-messages";
25
26// Fixed window: all executions in the same clock hour share one counter.
27export const RATE_LIMIT_WINDOW_MS = 3_600_000;
28// Calibration: the heaviest legitimate org runs ~1.1k executions per MONTH,
29// so 1000 per HOUR is far above any human-driven usage and only trips on
30// runaway automation (the incident this backstops: ~18k in 30 days would
31// still pass, which is fine — that class of overrun is the balance gate's
32// job; this catches tight loops).
33export const EXECUTIONS_PER_ORG_PER_HOUR = 1000;
34// Counter DO slower than this => fail open rather than stall executions.
35const RATE_LIMIT_CHECK_TIMEOUT_MS = 2_000;
36// The DO purges its storage this long after the last increment, so idle orgs
37// cost nothing. Two windows: long enough that an active window never purges.
38const COUNTER_PURGE_AFTER_MS = 2 * RATE_LIMIT_WINDOW_MS;
39
40export { RATE_LIMIT_BLOCKED_MESSAGE };
41
42export class ExecutionRateLimitExceededError extends Data.TaggedError(
43 "ExecutionRateLimitExceededError",
44)<{
45 readonly organizationId: string;
46 readonly message: string;
47}> {}
48
49/** Internal sentinel for a counter call that exceeded its time budget. */
50class RateLimitCheckTimeoutError extends Data.TaggedError("RateLimitCheckTimeoutError")<{
51 readonly timeoutMs: number;
52}> {}
53
54// ---------------------------------------------------------------------------
55// Counter Durable Object — one instance per organization (idFromName(orgId)).
56// Stores a single { windowId, count } record: an increment in a new window
57// resets the count, so old windows never accumulate. An alarm purges storage
58// after inactivity.
59// ---------------------------------------------------------------------------
60
61const WINDOW_RECORD_KEY = "window";
62
63type WindowRecord = {
64 readonly windowId: number;
65 readonly count: number;
66};
67
68export class ExecutionRateLimiterDO extends DurableObject {
69 private readonly counterStorage: DurableObjectState["storage"];
70
71 constructor(ctx: DurableObjectState, doEnv: Env) {
72 super(ctx, doEnv);
73 // Kept on an own field (not just inherited `this.ctx`) so tests can run
74 // the class against a fake storage under the `cloudflare:workers` stub.
75 this.counterStorage = ctx.storage;
76 }
77
78 /** Add one execution to `windowId`'s counter and return the new count. */
79 async increment(windowId: number): Promise<number> {
80 const stored = await this.counterStorage.get<WindowRecord>(WINDOW_RECORD_KEY);
81 const count = stored && stored.windowId === windowId ? stored.count + 1 : 1;
82 await this.counterStorage.put(WINDOW_RECORD_KEY, { windowId, count });
83 await this.counterStorage.setAlarm(Date.now() + COUNTER_PURGE_AFTER_MS);
84 return count;
85 }
86
87 async alarm(): Promise<void> {
88 await this.counterStorage.deleteAll();
89 }
90}
91
92// ---------------------------------------------------------------------------
93// Client
94// ---------------------------------------------------------------------------
95
96/** Count one execution for (organizationId, windowId); returns the new count. */
97export type RateLimitIncrement = (
98 organizationId: string,
99 windowId: number,
100) => Effect.Effect<number, unknown>;
101
102export type ExecutionRateLimiter = {
103 readonly decorate: <E extends Cause.YieldableError>(
104 organizationId: string,
105 engine: ExecutionEngine<E>,
106 ) => ExecutionEngine<E>;
107};
108
109/**
110 * Build a rate limiter around an increment function (in production: the
111 * counter DO). `options.limit` is the per-org hourly cap (production sets it
112 * from the env override in `makeCloudExecutionRateLimiter`); the rest tune the
113 * window and time budget.
114 */
115export const makeExecutionRateLimiter = (
116 increment: RateLimitIncrement,
117 options?: {
118 readonly limit?: number;
119 readonly windowMs?: number;
120 readonly timeoutMs?: number;
121 readonly now?: () => number;
122 },
123): ExecutionRateLimiter => {
124 const limit = options?.limit ?? EXECUTIONS_PER_ORG_PER_HOUR;
125 const windowMs = options?.windowMs ?? RATE_LIMIT_WINDOW_MS;
126 const timeoutMs = options?.timeoutMs ?? RATE_LIMIT_CHECK_TIMEOUT_MS;
127 const now = options?.now ?? Date.now;
128
129 const decide = (organizationId: string): Effect.Effect<GateDecision> =>
130 Effect.suspend(() => {
131 const windowId = Math.floor(now() / windowMs);
132 return increment(organizationId, windowId).pipe(
133 Effect.timeoutOrElse({
134 duration: `${timeoutMs} millis`,
135 orElse: () => Effect.fail(new RateLimitCheckTimeoutError({ timeoutMs })),
136 }),
137 Effect.map(
138 (count): GateDecision =>
139 count > limit
140 ? {
141 blocked: true,
142 error: new ExecutionRateLimitExceededError({
143 organizationId,
144 message: RATE_LIMIT_BLOCKED_MESSAGE,
145 }),
146 }
147 : { blocked: false },
148 ),
149 // FAIL OPEN: the backstop must never block executions because its
150 // counter is unreachable or slow.
151 Effect.catch((error: unknown) =>
152 Effect.sync((): GateDecision => {
153 console.warn("[rate-limit] execution rate limit check failed open:", error);
154 Sentry.captureException(error);
155 return { blocked: false };
156 }),
157 ),
158 );
159 });
160
161 return {
162 decorate: (organizationId, engine) => withPreExecutionGate(engine, decide(organizationId)),
163 };
164};
165
166// ---------------------------------------------------------------------------
167// Cloud wiring — reads the EXECUTION_RATE_LIMITER binding from the worker env.
168// ---------------------------------------------------------------------------
169
170// The DO stub's RPC surface. The binding is declared untyped in
171// env-augment.d.ts (matching the BLOBS precedent), so the call site narrows it
172// to the one method the class exposes.
173type ExecutionRateLimiterStub = {
174 readonly increment: (windowId: number) => Promise<number>;
175};
176
177type RateLimiterNamespace = {
178 readonly idFromName: (name: string) => DurableObjectId;
179 readonly get: (id: DurableObjectId) => unknown;
180};
181
182/**
183 * Production rate limiter backed by the `EXECUTION_RATE_LIMITER` counter DO.
184 * When the binding is absent (unit-test workers, older local setups) the
185 * limiter is disabled: every check passes, logged once at construction.
186 */
187export const makeCloudExecutionRateLimiter = (): ExecutionRateLimiter => {
188 const limit = resolveRateLimit();
189 const namespace = (env as { EXECUTION_RATE_LIMITER?: RateLimiterNamespace })
190 .EXECUTION_RATE_LIMITER;
191 if (!namespace) {
192 console.warn(
193 "[rate-limit] EXECUTION_RATE_LIMITER binding missing; execution rate limiting disabled",
194 );
195 return makeExecutionRateLimiter(() => Effect.succeed(0));
196 }
197 return makeExecutionRateLimiter(
198 (organizationId, windowId) =>
199 Effect.tryPromise(() => {
200 const stub = namespace.get(
201 namespace.idFromName(organizationId),
202 ) as ExecutionRateLimiterStub;
203 return stub.increment(windowId);
204 }),
205 { limit },
206 );
207};
208
209/**
210 * The per-org hourly cap: the `EXECUTION_RATE_LIMIT_PER_HOUR` env override
211 * (parsed as a positive integer) or `EXECUTIONS_PER_ORG_PER_HOUR` when it's
212 * unset or unparseable. The override exists so e2e can drive the backstop with
213 * a small number of real executions; production leaves the var unset.
214 */
215const resolveRateLimit = (): number => {
216 const raw = (env as { EXECUTION_RATE_LIMIT_PER_HOUR?: string }).EXECUTION_RATE_LIMIT_PER_HOUR;
217 if (raw === undefined) return EXECUTIONS_PER_ORG_PER_HOUR;
218 const parsed = Number.parseInt(raw, 10);
219 return Number.isInteger(parsed) && parsed > 0 ? parsed : EXECUTIONS_PER_ORG_PER_HOUR;
220};
apps/cloud/src/engine/execution-stack-metered.ts+30−9
@@ -28,19 +28,40 @@ import { · 27 unmodified lines
28import { AutumnService } from "../extensions/billing/service";28import { AutumnService } from "../extensions/billing/service";
29import type { DbService } from "../db/db";29import type { DbService } from "../db/db";
30import { CloudExecutionSeamsLayer } from "../engine/execution-stack";30import { CloudExecutionSeamsLayer } from "../engine/execution-stack";
31import { makeExecutionLimitGate } from "./execution-gate";
32import { makeCloudExecutionRateLimiter } from "./execution-rate-limit";
31import { withExecutionUsageTracking } from "./execution-usage";33import { withExecutionUsageTracking } from "./execution-usage";
32 34
33// Usage-metering decorator bound to the billing service. `trackExecution` is35// Usage-metering decorator bound to the billing service, plus the two
34// fire-and-forget (`Effect.runFork`) so the billing call can't stall a36// pre-execution guards this layer owns, ordered cheapest first:
35// user-facing execution.37// 1. rate-limit backstop (counter DO, independent of billing)
38// 2. execution balance gate (Autumn check, cached 60s, fails open)
39// 3. usage tracking — fire-and-forget (`Effect.runFork`) so the billing
40// call can't stall a user-facing execution.
41// The guards wrap OUTSIDE the tracker so a blocked execution is neither run
42// nor tracked. One gate/limiter instance per layer build: the balance cache is
43// shared by every engine this decorator produces (in the MCP session DO that's
44// the session's engines; on the HTTP plane the boot layer builds it once).
36export const CloudMeteringEngineDecorator: Layer.Layer<EngineDecorator, never, AutumnService> =45export const CloudMeteringEngineDecorator: Layer.Layer<EngineDecorator, never, AutumnService> =
37 Layer.effect(EngineDecorator)(46 Layer.effect(EngineDecorator)(
38 Effect.map(AutumnService.asEffect(), (autumn): EngineDecorator["Service"] => ({47 Effect.map(AutumnService.asEffect(), (autumn): EngineDecorator["Service"] => {
39 decorate: (engine, identity: EngineStackIdentity) =>48 const balanceGate = makeExecutionLimitGate((organizationId) =>
40 withExecutionUsageTracking(identity.organizationId, engine, (organizationId) =>49 autumn.checkExecutionBalance(organizationId),
41 Effect.runFork(autumn.trackExecution(organizationId)),50 );
42 ),51 const rateLimiter = makeCloudExecutionRateLimiter();
43 })),52 return {
53 decorate: (engine, identity: EngineStackIdentity) =>
54 rateLimiter.decorate(
55 identity.organizationId,
56 balanceGate.decorate(
57 identity.organizationId,
58 withExecutionUsageTracking(identity.organizationId, engine, (organizationId) =>
59 Effect.runFork(autumn.trackExecution(organizationId)),
60 ),
61 ),
62 ),
63 };
64 }),
44 );65 );
45 66
46/**67/**
apps/cloud/src/env-augment.d.ts+14−0
@@ -32,6 +32,20 @@ declare global { · 31 unmodified lines
32 // ON; the test workers set "true" so fixtures can reach localhost.32 // ON; the test workers set "true" so fixtures can reach localhost.
33 ALLOW_LOCAL_NETWORK?: string;33 ALLOW_LOCAL_NETWORK?: string;
34 34
35 // Per-org execution rate-limit counter DO (wrangler.jsonc
36 // `durable_objects`). Declared optional here (matching the BLOBS
37 // precedent) rather than regenerating worker-configuration.d.ts: test
38 // workers and older local setups run without the binding, and the
39 // limiter degrades to disabled when absent.
40 EXECUTION_RATE_LIMITER?: import("@cloudflare/workers-types").DurableObjectNamespace;
41
42 // Optional per-org hourly execution rate-limit override, parsed as an
43 // integer (defaults to EXECUTIONS_PER_ORG_PER_HOUR = 1000 when unset or
44 // unparseable). Exists for e2e: the production cap of 1000/hour can't be
45 // exercised with real executions, so the e2e dev-server env sets a small
46 // number to drive the backstop. Production leaves it unset.
47 EXECUTION_RATE_LIMIT_PER_HOUR?: string;
48
35 // Billing49 // Billing
36 AUTUMN_SECRET_KEY?: string;50 AUTUMN_SECRET_KEY?: string;
37 /** Optional Autumn base-URL override (Autumn emulator in tests/dev). */51 /** Optional Autumn base-URL override (Autumn emulator in tests/dev). */
apps/cloud/src/extensions/billing/service.ts+14−1
@@ -22,6 +22,9 @@ export class AutumnError extends Data.TaggedError("AutumnError")<{ · 21 unmodified lines
22 22
23export type IAutumnService = Readonly<{23export type IAutumnService = Readonly<{
24 use: <A>(fn: (client: Autumn) => Promise<A>) => Effect.Effect<A, AutumnError, never>;24 use: <A>(fn: (client: Autumn) => Promise<A>) => Effect.Effect<A, AutumnError, never>;
25 checkExecutionBalance: (
26 organizationId: string,
27 ) => Effect.Effect<{ readonly allowed: boolean }, AutumnError, never>;
25 /**28 /**
26 * Fire-and-forget-safe execution usage tracker. Errors are caught and29 * Fire-and-forget-safe execution usage tracker. Errors are caught and
27 * logged; the returned Effect never fails. Callers typically30 * logged; the returned Effect never fails. Callers typically
@@ -44,6 +47,7 @@ const make = Effect.sync(() => { · 16 unmodified lines
44 );47 );
45 return {48 return {
46 use: () => notConfigured,49 use: () => notConfigured,
50 checkExecutionBalance: () => notConfigured,
47 trackExecution: () => Effect.void,51 trackExecution: () => Effect.void,
48 } satisfies IAutumnService;52 } satisfies IAutumnService;
49 }53 }
@@ -80,7 +84,16 @@ const make = Effect.sync(() => { · 30 unmodified lines
80 );84 );
81 }).pipe(Effect.withSpan("autumn.trackExecution"));85 }).pipe(Effect.withSpan("autumn.trackExecution"));
82 86
83 return { use, trackExecution } satisfies IAutumnService;87 const checkExecutionBalance = (organizationId: string) =>
88 Effect.gen(function* () {
89 yield* Effect.annotateCurrentSpan({ "autumn.customer.id": organizationId });
90 const check = yield* use((c) =>
91 c.check({ customerId: organizationId, featureId: "executions" }),
92 );
93 return { allowed: check.allowed };
94 }).pipe(Effect.withSpan("autumn.checkExecutionBalance"));
95
96 return { use, checkExecutionBalance, trackExecution } satisfies IAutumnService;
84});97});
85 98
86export class AutumnService extends Context.Service<AutumnService, IAutumnService>()(99export class AutumnService extends Context.Service<AutumnService, IAutumnService>()(
apps/cloud/src/server.ts+5−0
@@ -55,6 +55,11 @@ export const McpSessionDOSqlite = Sentry.instrumentDurableObjectWithSentry( · 54 unmodified lines
55// (with a `deleted_classes: ["McpSessionDO"]` migration) now that nothing binds it.55// (with a `deleted_classes: ["McpSessionDO"]` migration) now that nothing binds it.
56export class McpSessionDO extends DurableObject {}56export class McpSessionDO extends DurableObject {}
57 57
58// Per-org execution rate-limit counter DO (abuse backstop; migration v3,
59// `EXECUTION_RATE_LIMITER` binding). Plain counter, no Sentry wrapper needed:
60// its callers already fail open and report errors themselves.
61export { ExecutionRateLimiterDO } from "./engine/execution-rate-limit";
62
58// ---------------------------------------------------------------------------63// ---------------------------------------------------------------------------
59// Worker fetch handler64// Worker fetch handler
60//65//
apps/cloud/wrangler.jsonc+11−0
@@ -22,6 +22,13 @@ · 21 unmodified lines
22 "name": "MCP_SESSION",22 "name": "MCP_SESSION",
23 "class_name": "McpSessionDOSqlite",23 "class_name": "McpSessionDOSqlite",
24 },24 },
25 // Per-org execution rate-limit counter (abuse backstop). One instance
26 // per organization via idFromName(orgId); a single { windowId, count }
27 // record per instance, purged by alarm after inactivity.
28 {
29 "name": "EXECUTION_RATE_LIMITER",
30 "class_name": "ExecutionRateLimiterDO",
31 },
25 ],32 ],
26 },33 },
27 // The MCP session DO moved to the Cloudflare Agents (`McpAgent`) base, which34 // The MCP session DO moved to the Cloudflare Agents (`McpAgent`) base, which
@@ -42,6 +49,10 @@ · 14 unmodified lines
42 "tag": "v2",49 "tag": "v2",
43 "new_sqlite_classes": ["McpSessionDOSqlite"],50 "new_sqlite_classes": ["McpSessionDOSqlite"],
44 },51 },
52 {
53 "tag": "v3",
54 "new_sqlite_classes": ["ExecutionRateLimiterDO"],
55 },
45 ],56 ],
46 "services": [57 "services": [
47 {58 {
bun.lock+2−2
@@ -344,7 +344,7 @@ · 343 unmodified lines
344 "version": "0.0.24",344 "version": "0.0.24",
345 "dependencies": {345 "dependencies": {
346 "@executor-js/api": "workspace:*",346 "@executor-js/api": "workspace:*",
347 "@executor-js/emulate": "^0.10.0",347 "@executor-js/emulate": "^0.13.2",
348 "@executor-js/mcporter": "^0.11.4",348 "@executor-js/mcporter": "^0.11.4",
349 "@executor-js/plugin-graphql": "workspace:*",349 "@executor-js/plugin-graphql": "workspace:*",
350 "@executor-js/plugin-mcp": "workspace:*",350 "@executor-js/plugin-mcp": "workspace:*",
@@ -1743,7 +1743,7 @@ · 1392 unmodified lines
1743 1743
1744 "@executor-js/e2e": ["@executor-js/e2e@workspace:e2e"],1744 "@executor-js/e2e": ["@executor-js/e2e@workspace:e2e"],
1745 1745
1746 "@executor-js/emulate": ["@executor-js/emulate@0.10.0", "", { "dependencies": { "@aws-sdk/client-s3": "^3.1031.0", "@aws-sdk/client-sqs": "^3.1075.0", "@azure/msal-node": "^5.3.0", "@clerk/backend": "^3.8.4", "@octokit/rest": "^22.0.1", "@okta/okta-auth-js": "^8.0.1", "@slack/web-api": "^7.16.0", "@vercel/sdk": "^1.28.4", "@workos-inc/node": "^8.13.0", "atlas-api-client": "^0.3.0", "autumn-js": "^1.2.8", "commander": "^14", "googleapis": "^173.0.0", "graphql": "^16.9.0", "graphql-request": "^7.4.0", "openid-client": "^6.8.4", "picocolors": "^1.1.1", "resend": "^6.16.0", "spotify-web-api-node": "^5.0.2", "stripe": "^22.3.0", "twitter-api-v2": "^1.29.0", "yaml": "^2" }, "bin": { "emulate": "dist/index.js" } }, "sha512-GE1+XDQ4FJt4ZDrwNjuUqUEG1WaH06UE12ME/xJdcCNbsa6EE6SA+i8onVJQ5Dr7DbUveU37E4djLTiAcceLPw=="],1746 "@executor-js/emulate": ["@executor-js/emulate@0.13.2", "", { "dependencies": { "@aws-sdk/client-s3": "^3.1031.0", "@aws-sdk/client-sqs": "^3.1075.0", "@azure/msal-node": "^5.3.0", "@clerk/backend": "^3.8.4", "@octokit/rest": "^22.0.1", "@okta/okta-auth-js": "^8.0.1", "@slack/web-api": "^7.16.0", "@vercel/sdk": "^1.28.4", "@workos-inc/node": "^8.13.0", "atlas-api-client": "^0.3.0", "autumn-js": "^1.2.8", "commander": "^14", "googleapis": "^173.0.0", "graphql": "^16.9.0", "graphql-request": "^7.4.0", "openid-client": "^6.8.4", "picocolors": "^1.1.1", "resend": "^6.16.0", "spotify-web-api-node": "^5.0.2", "stripe": "^22.3.0", "twitter-api-v2": "^1.29.0", "yaml": "^2" }, "bin": { "emulate": "dist/index.js" } }, "sha512-ikmzZgBrRnmJgKKOW9TE2KCCdenKtihx2nWW/mMMmugn3UlVJsxuIxp6jv0BEo4GZcY8Ve2oka6UEKl+gBKRZg=="],
1747 1747
1748 "@executor-js/example-all-plugins": ["@executor-js/example-all-plugins@workspace:examples/all-plugins"],1748 "@executor-js/example-all-plugins": ["@executor-js/example-all-plugins@workspace:examples/all-plugins"],
1749 1749
e2e/cloud/mcp-execution-limits.test.ts+200−0
@@ -0,0 +1,200 @@
1// Cloud-only (billing): the pre-execution guards that sit in front of the MCP
2// session Durable Object's execution engine. Two independent gates run before
3// every new `execute`:
4//
5// - the BALANCE gate consults Autumn's `balances.check` and blocks once an
6// org has used its plan's included executions, and
7// - the RATE-LIMIT backstop counts executions per org per hour and blocks
8// runaway automation even when billing is down.
9//
10// Both FAIL OPEN: an Autumn error or timeout must never take executions down
11// with the billing provider. These scenarios pin all four behaviours end to
12// end against the real workerd + McpSessionDO topology, driving a real MCP
13// client and reading Autumn's own ledger as ground truth — a blocked execution
14// must not reach the meter, and a failed-open one must show the check was
15// actually attempted (and faulted) rather than silently skipped.
16import { expect } from "@effect/vitest";
17import { Effect } from "effect";
18
19import {
20 EXECUTION_LIMIT_BLOCKED_MESSAGE,
21 RATE_LIMIT_BLOCKED_MESSAGE,
22} from "../../apps/cloud/src/engine/execution-limit-messages";
23import { scenario } from "../src/scenario";
24import { Autumn, Billing, Mcp, Target } from "../src/services";
25import { E2E_EXECUTION_RATE_LIMIT } from "../setup/execution-limits";
26import type { Identity } from "../src/target";
27
28const emailOf = (identity: Identity): string => identity.credentials?.email ?? identity.label;
29
30/** The org the bearer is scoped to — the Autumn customer id the guards decide
31 * against — read from the JWT's public claims. */
32const orgIdOf = (bearer: string): string => {
33 const claims = JSON.parse(Buffer.from(bearer.split(".")[1] ?? "", "base64url").toString()) as {
34 readonly org_id?: string;
35 };
36 if (!claims.org_id) throw new Error("orgIdOf: bearer carries no org_id claim");
37 return claims.org_id;
38};
39
40// The e2e worker is booted with a small EXECUTION_RATE_LIMIT_PER_HOUR so the
41// backstop is reachable with real executions (cloud.boot.ts). The prod cap of
42// 1000/hour can't be. One shared constant (setup/execution-limits.ts) keeps
43// this and the boot env from drifting apart.
44const RATE_LIMIT = E2E_EXECUTION_RATE_LIMIT;
45
46scenario(
47 "Billing · an org out of executions is blocked before the code ever runs",
48 { timeout: 180_000 },
49 Effect.gen(function* () {
50 // Gates: billing is enforced here AND the Autumn ledger is observable.
51 // Yield before any work so a target missing either capability skips cleanly.
52 yield* Billing;
53 const autumn = yield* Autumn;
54 const target = yield* Target;
55 const mcp = yield* Mcp;
56
57 const identity = yield* target.newIdentity();
58 const bearer = yield* mcp.mintBearer(emailOf(identity));
59 const customerId = orgIdOf(bearer);
60
61 // Burn the whole included allotment so the org's remaining balance is zero.
62 // This records one `balances.track` event (the plan's included amount) —
63 // the baseline the blocked execution is measured against below.
64 yield* autumn.exhaustExecutions(customerId);
65 const afterExhaust = yield* autumn.usageEvents({ customerId, featureId: "executions" });
66 expect(afterExhaust.length, "exhausting the balance records exactly one usage event").toBe(1);
67
68 // A NEW MCP session => a fresh session DO => a fresh gate cache, so the
69 // balance is checked live (no 60s stale-allow window from a prior session).
70 const session = mcp.session(identity);
71 const result = yield* session.call("execute", { code: "return 1 + 1;" });
72
73 expect(result.ok, "a blocked execution surfaces as an MCP error result").toBe(false);
74 expect(result.text, "the client sees the execution-limit message").toContain(
75 EXECUTION_LIMIT_BLOCKED_MESSAGE,
76 );
77
78 // The meter is the source of truth: a blocked execution is neither run nor
79 // usage-tracked. Give any (erroneous) fire-and-forget track ample time to
80 // land, then assert nothing beyond the exhaust event was recorded.
81 const settle = yield* autumn.usageEvents({ customerId, featureId: "executions" });
82 expect(settle.length, "the blocked execution adds no 'executions' usage event").toBe(
83 afterExhaust.length,
84 );
85 }),
86);
87
88scenario(
89 "Billing · a balance-check failure fails open and still runs the execution",
90 { timeout: 180_000 },
91 Effect.gen(function* () {
92 yield* Billing;
93 const autumn = yield* Autumn;
94 const target = yield* Target;
95 const mcp = yield* Mcp;
96
97 const identity = yield* target.newIdentity();
98
99 yield* Effect.gen(function* () {
100 // One-shot 500 on the very next balances.check the gate makes.
101 yield* autumn.armFault({
102 match: { operationId: "balances.check" },
103 response: { status: 500, body: { message: "emulated billing outage" } },
104 times: 1,
105 });
106
107 const session = mcp.session(identity);
108 const result = yield* session.call("execute", { code: "return 6 * 7;" });
109
110 // Fail open: the outage must not block a solvent org's execution.
111 expect(result.ok, "the execution succeeds despite the billing outage").toBe(true);
112 expect(result.text, "it returns its value").toContain("42");
113
114 // Proof the gate actually CONSULTED Autumn and failed open (rather than
115 // never checking): the faulted balances.check is in the emulator ledger.
116 const checks = yield* autumn.ledgerFor("balances.check");
117 expect(
118 checks.some((entry) => entry.faulted),
119 "the gate's balances.check reached Autumn and was faulted",
120 ).toBe(true);
121 }).pipe(Effect.ensuring(autumn.clearFaults().pipe(Effect.ignore)));
122 }),
123);
124
125scenario(
126 "Billing · a balance-check timeout fails open and still runs the execution",
127 { timeout: 180_000 },
128 Effect.gen(function* () {
129 yield* Billing;
130 const autumn = yield* Autumn;
131 const target = yield* Target;
132 const mcp = yield* Mcp;
133
134 const identity = yield* target.newIdentity();
135
136 yield* Effect.gen(function* () {
137 // The armed check stalls 3s before returning — past the gate's 2s budget,
138 // so the gate times out client-side and fails open. (The response still
139 // arrives after the delay; the ~2s added latency fits the timeout.)
140 yield* autumn.armFault({
141 match: { operationId: "balances.check" },
142 response: { status: 200, body: { allowed: true } },
143 times: 1,
144 delayMs: 3000,
145 });
146
147 const session = mcp.session(identity);
148 const result = yield* session.call("execute", { code: "return 21 * 2;" });
149
150 expect(result.ok, "the execution succeeds despite the slow balance check").toBe(true);
151 expect(result.text, "it returns its value").toContain("42");
152
153 const checks = yield* autumn.ledgerFor("balances.check");
154 expect(checks.length, "the gate did attempt the (stalled) balances.check").toBeGreaterThan(0);
155 }).pipe(Effect.ensuring(autumn.clearFaults().pipe(Effect.ignore)));
156 }),
157);
158
159scenario(
160 "Billing · the rate-limit backstop blocks runaway executions and doesn't meter them",
161 { timeout: 180_000 },
162 Effect.gen(function* () {
163 yield* Billing;
164 const autumn = yield* Autumn;
165 const target = yield* Target;
166 const mcp = yield* Mcp;
167
168 const identity = yield* target.newIdentity();
169 const bearer = yield* mcp.mintBearer(emailOf(identity));
170 const customerId = orgIdOf(bearer);
171
172 // All executions run in ONE session so the balance cache stays allowed
173 // (the org is solvent) — only the per-org hourly counter should trip.
174 const session = mcp.session(identity);
175
176 // Run exactly the limit's worth of allowed executions.
177 for (let i = 1; i <= RATE_LIMIT; i++) {
178 const ok = yield* session.call("execute", { code: `return ${i};` });
179 expect(ok.ok, `execution ${i} (within the limit) succeeds`).toBe(true);
180 }
181
182 // The next one crosses the hourly cap: blocked with the backstop message.
183 const blocked = yield* session.call("execute", { code: "return 999;" });
184 expect(blocked.ok, "the execution past the cap is an MCP error result").toBe(false);
185 expect(blocked.text, "the client sees the rate-limit message").toContain(
186 RATE_LIMIT_BLOCKED_MESSAGE,
187 );
188
189 // A rate-limited execution is neither run nor metered: exactly the allowed
190 // runs reached the Autumn meter, and the blocked one added nothing.
191 const metered = yield* autumn.expectUsage({
192 customerId,
193 featureId: "executions",
194 count: RATE_LIMIT,
195 });
196 expect(metered.length, "only the allowed executions are metered — the blocked one is not").toBe(
197 RATE_LIMIT,
198 );
199 }),
200);
e2e/package.json+1−1
@@ -22,7 +22,7 @@ · 21 unmodified lines
22 },22 },
23 "dependencies": {23 "dependencies": {
24 "@executor-js/api": "workspace:*",24 "@executor-js/api": "workspace:*",
25 "@executor-js/emulate": "^0.10.0",25 "@executor-js/emulate": "^0.13.2",
26 "@executor-js/mcporter": "^0.11.4",26 "@executor-js/mcporter": "^0.11.4",
27 "@executor-js/plugin-graphql": "workspace:*",27 "@executor-js/plugin-graphql": "workspace:*",
28 "@executor-js/plugin-mcp": "workspace:*",28 "@executor-js/plugin-mcp": "workspace:*",
e2e/setup/autumn-plans.ts+18−0
@@ -49,3 +49,21 @@ const toSeed = (plan: AtmnPlan): AutumnSeedPlan => ({ · 48 unmodified lines
49export const AUTUMN_PLAN_SEED: AutumnSeedPlan[] = [free, team, enterprise].map((p) =>49export const AUTUMN_PLAN_SEED: AutumnSeedPlan[] = [free, team, enterprise].map((p) =>
50 toSeed(p as AtmnPlan),50 toSeed(p as AtmnPlan),
51);51);
52
53/**
54 * The "executions" allotment a fresh org has on the auto-enabled default plan
55 * (Free): the exact number of executions it takes to exhaust the quota. Read
56 * from the same seed the emulator is booted with, so a plan change in
57 * autumn.config.ts flows straight into the balance-gate scenarios rather than
58 * drifting against a hardcoded magic number.
59 */
60export const DEFAULT_PLAN_EXECUTIONS_INCLUDED: number = (() => {
61 const defaultPlan = AUTUMN_PLAN_SEED.find((plan) => plan.auto_enable);
62 const executions = defaultPlan?.items.find((item) => item.feature_id === "executions");
63 if (executions?.included === undefined) {
64 throw new Error(
65 "autumn-plans: the auto-enabled default plan has no included 'executions' amount to exhaust",
66 );
67 }
68 return executions.included;
69})();
e2e/setup/cloud.boot.ts+7−0
@@ -12,6 +12,7 @@ import { createEmulator } from "@executor-js/emulate"; · 11 unmodified lines
12 12
13import { bootProcesses, waitForHttp } from "./boot";13import { bootProcesses, waitForHttp } from "./boot";
14import { AUTUMN_PLAN_SEED } from "./autumn-plans";14import { AUTUMN_PLAN_SEED } from "./autumn-plans";
15import { E2E_EXECUTION_RATE_LIMIT } from "./execution-limits";
15 16
16export const cloudDir = fileURLToPath(new URL("../../apps/cloud/", import.meta.url));17export const cloudDir = fileURLToPath(new URL("../../apps/cloud/", import.meta.url));
17 18
@@ -90,6 +91,12 @@ export const bootCloud = async (options: CloudBootOptions): Promise<CloudBooted> · 72 unmodified lines
90 MCP_AUTHKIT_DOMAIN: workosUrl,91 MCP_AUTHKIT_DOMAIN: workosUrl,
91 MCP_RESOURCE_ORIGIN: options.publicUrl,92 MCP_RESOURCE_ORIGIN: options.publicUrl,
92 ALLOW_LOCAL_NETWORK: "true",93 ALLOW_LOCAL_NETWORK: "true",
94 // Shrink the per-org hourly execution cap (prod default 1000) to a number
95 // the rate-limit-backstop scenario can actually exhaust with real
96 // executions — but see execution-limits.ts: it must stay above every other
97 // scenario's per-org execute count. Reaches the worker via
98 // CLOUDFLARE_INCLUDE_PROCESS_ENV, same as ALLOW_LOCAL_NETWORK.
99 EXECUTION_RATE_LIMIT_PER_HOUR: String(E2E_EXECUTION_RATE_LIMIT),
93 // Throwaway PGlite on its own port + dir so it never fights `bun dev`.100 // Throwaway PGlite on its own port + dir so it never fights `bun dev`.
94 DEV_DB_PORT: String(options.dbPort),101 DEV_DB_PORT: String(options.dbPort),
95 DEV_DB_PATH: dbPath,102 DEV_DB_PATH: dbPath,
e2e/setup/execution-limits.ts+14−0
@@ -0,0 +1,14 @@
1// The e2e worker's per-org hourly execution cap (EXECUTION_RATE_LIMIT_PER_HOUR).
2// One constant with two consumers, the boot recipe env (cloud.boot.ts) and the
3// rate-limit backstop scenario (cloud/mcp-execution-limits.test.ts), so they
4// cannot drift apart.
5//
6// Picking the value is a squeeze from both sides. It must be LOW enough that
7// the backstop scenario can exhaust it with real sequential executions (prod's
8// 1000/hour cannot be reached in a test), and HIGH enough that no other
9// scenario trips it: the counter is per organization and every `execute` a
10// scenario runs counts against its org, so this must exceed the busiest
11// single-org scenario's execute count (currently toolkits-mcp at ~8) with
12// comfortable headroom. If a scenario ever fails with the rate-limit backstop
13// message, it outgrew this cap: raise it here, never in the boot env alone.
14export const E2E_EXECUTION_RATE_LIMIT = 20;
e2e/src/surfaces/autumn.ts+123−0
@@ -9,6 +9,8 @@ · 8 unmodified lines
9// so the contract has to be pinned where the usage is read.9// so the contract has to be pinned where the usage is read.
10import { Effect, Schedule } from "effect";10import { Effect, Schedule } from "effect";
11 11
12import { DEFAULT_PLAN_EXECUTIONS_INCLUDED } from "../../setup/autumn-plans";
13
12/** One usage event Autumn recorded: a `track` of `value` units of `featureId`14/** One usage event Autumn recorded: a `track` of `value` units of `featureId`
13 * for `customerId` (the organization the execution ran under). */15 * for `customerId` (the organization the execution ran under). */
14export interface UsageEvent {16export interface UsageEvent {
@@ -24,6 +26,33 @@ export interface UsageQuery { · 9 unmodified lines
24 readonly featureId: string;26 readonly featureId: string;
25}27}
26 28
29/** A one-shot (or counted) fault to arm against a matching Autumn request. Mirrors
30 * the emulator's `POST /_emulate/faults` body: a request matching every provided
31 * `match` criterion short-circuits with `response`, `times` decrements per hit
32 * (default 1, removed at zero), and `delayMs` stalls before the fault returns —
33 * the knob a timeout scenario uses to push Autumn past the gate's time budget. */
34export interface FaultInput {
35 readonly match: {
36 readonly operationId?: string;
37 readonly method?: string;
38 readonly pathPattern?: string;
39 };
40 readonly response: { readonly status: number; readonly body?: unknown };
41 readonly times?: number;
42 readonly delayMs?: number;
43}
44
45/** One request the emulator recorded in its ledger, narrowed to what these
46 * scenarios assert on: which operation ran and whether a fault short-circuited
47 * it. A faulted `balances.check` entry is the proof the gate actually consulted
48 * Autumn (and then failed open) rather than never checking at all. */
49export interface LedgerEntry {
50 readonly operationId?: string;
51 readonly method: string;
52 readonly path: string;
53 readonly faulted: boolean;
54}
55
27export interface AutumnSurface {56export interface AutumnSurface {
28 /** One-shot read of the matching usage events from the ledger. */57 /** One-shot read of the matching usage events from the ledger. */
29 readonly usageEvents: (query: UsageQuery) => Effect.Effect<readonly UsageEvent[], unknown>;58 readonly usageEvents: (query: UsageQuery) => Effect.Effect<readonly UsageEvent[], unknown>;
@@ -41,6 +70,20 @@ export interface AutumnSurface { · 11 unmodified lines
41 * the billing backend becomes consistent. The `sessionId` is the last path70 * the billing backend becomes consistent. The `sessionId` is the last path
42 * segment of the hosted checkout URL the browser was sent to. */71 * segment of the hosted checkout URL the browser was sent to. */
43 readonly settleCheckout: (sessionId: string) => Effect.Effect<void, unknown>;72 readonly settleCheckout: (sessionId: string) => Effect.Effect<void, unknown>;
73 /** Burn an org's entire remaining "executions" balance in one `balances.track`,
74 * so the next `balances.check` reports `allowed: false`. The amount is the
75 * default plan's included allotment (read from the plan seed, never hardcoded),
76 * which drives `remaining` to zero. This is the setup a "blocked at cap"
77 * scenario runs before opening the session it expects to be gated. */
78 readonly exhaustExecutions: (customerId: string) => Effect.Effect<void, unknown>;
79 /** Arm a fault against a matching Autumn request (`POST /_emulate/faults`). */
80 readonly armFault: (input: FaultInput) => Effect.Effect<void, unknown>;
81 /** Clear every armed fault (`DELETE /_emulate/faults`) — the finalizer that
82 * keeps a scenario's fault from leaking into the next one. */
83 readonly clearFaults: () => Effect.Effect<void, unknown>;
84 /** Read the emulator's request ledger, filtered to one operation — used to
85 * prove a faulted `balances.check` was actually attempted (fail-open proof). */
86 readonly ledgerFor: (operationId: string) => Effect.Effect<readonly LedgerEntry[], unknown>;
44}87}
45 88
46export const makeAutumnSurface = (autumnUrl: string): AutumnSurface => {89export const makeAutumnSurface = (autumnUrl: string): AutumnSurface => {
@@ -90,9 +133,89 @@ export const makeAutumnSurface = (autumnUrl: string): AutumnSurface => { · 43 unmodified lines
90 }133 }
91 });134 });
92 135
136 // Track exactly the plan's included allotment in one event, driving
137 // `remaining` (included - usage) to zero so the next check reports blocked.
138 const exhaustExecutions = (customerId: string) =>
139 Effect.gen(function* () {
140 const response = yield* Effect.promise(() =>
141 fetch(`${autumnUrl}/v1/balances.track`, {
142 method: "POST",
143 headers: { "content-type": "application/json" },
144 body: JSON.stringify({
145 customer_id: customerId,
146 feature_id: "executions",
147 value: DEFAULT_PLAN_EXECUTIONS_INCLUDED,
148 }),
149 }),
150 );
151 if (!response.ok) {
152 return yield* Effect.fail(
153 `autumn balances.track responded ${response.status}: ${yield* Effect.promise(() => response.text())}`,
154 );
155 }
156 });
157
158 const armFault = (input: FaultInput) =>
159 Effect.gen(function* () {
160 const response = yield* Effect.promise(() =>
161 fetch(`${autumnUrl}/_emulate/faults`, {
162 method: "POST",
163 headers: { "content-type": "application/json" },
164 body: JSON.stringify(input),
165 }),
166 );
167 if (!response.ok) {
168 return yield* Effect.fail(
169 `autumn arm fault responded ${response.status}: ${yield* Effect.promise(() => response.text())}`,
170 );
171 }
172 });
173
174 const clearFaults = () =>
175 Effect.gen(function* () {
176 const response = yield* Effect.promise(() =>
177 fetch(`${autumnUrl}/_emulate/faults`, { method: "DELETE" }),
178 );
179 if (!response.ok) {
180 return yield* Effect.fail(
181 `autumn clear faults responded ${response.status}: ${yield* Effect.promise(() => response.text())}`,
182 );
183 }
184 });
185
186 const ledgerFor = (operationId: string) =>
187 Effect.gen(function* () {
188 const response = yield* Effect.promise(() => fetch(`${autumnUrl}/_emulate/ledger`));
189 if (!response.ok) {
190 return yield* Effect.fail(
191 `autumn ledger responded ${response.status}: ${yield* Effect.promise(() => response.text())}`,
192 );
193 }
194 const body = (yield* Effect.promise(() => response.json())) as {
195 readonly entries?: ReadonlyArray<{
196 readonly operationId?: string;
197 readonly method?: string;
198 readonly path?: string;
199 readonly faulted?: boolean;
200 }>;
201 };
202 return (body.entries ?? [])
203 .filter((entry) => entry.operationId === operationId)
204 .map((entry) => ({
205 operationId: entry.operationId,
206 method: entry.method ?? "",
207 path: entry.path ?? "",
208 faulted: entry.faulted === true,
209 }));
210 });
211
93 return {212 return {
94 usageEvents,213 usageEvents,
95 settleCheckout,214 settleCheckout,
215 exhaustExecutions,
216 armFault,
217 clearFaults,
218 ledgerFor,
96 expectUsage: (query) =>219 expectUsage: (query) =>
97 usageEvents(query).pipe(220 usageEvents(query).pipe(
98 Effect.filterOrFail(221 Effect.filterOrFail(