History

openapi: stream large specs so the full Microsoft Graph spec fits the Workers isolate (#1091)

* openapi: stream compile + serve from content-addressed bindings (no re-parse) The serve path (resolveOpenApiBackedTools, the 2nd OOM site) re-parsed the whole spec on every tools/list. For the full Microsoft Graph spec (37MB / 16.5k ops) that rebuilt a ~300MB document tree and OOM'd the 128MB Workers isolate even after the add-path parser swap. Add a memory-safe compile + serve path that never re-parses: - streamOperationBindings (extract.ts): two-pass streaming compile. Pass 1 plans tool paths from schema-free metadata (planToolPaths, extracted from compileToolDefinitions); pass 2 builds invocation bindings in bounded chunks, flushing each before the next so a huge spec's bindings are never all co-resident with the parsed tree. - Content-addressed defs blob (store.ts: putDefs/getDefs, keyed by specHash): the normalized #/$defs schemas, built once per spec at add time and shared across tenants. buildDefsJson serializes them one schema at a time so the normalized tree never co-resides with the parsed document. - Persist a resolved description per operation so the serve path rebuilds each tool def from the binding alone. - resolveOpenApiBackedTools fast path: getDefs(specHash) + listOperations + toolDefFromStoredOperation, falling back to a spec re-parse for legacy rows or a missing/corrupt blob. - Route the openapi, microsoft (full-graph), and google add/update paths through the streaming persist + putDefs. * e2e: guard full Graph catalog add + serve at real scale Drives the public API to add every Microsoft Graph workload (the full ~37MB / 16.5k-operation spec) and then list the served catalog. Pins both former OOM sites at real scale: the streaming add persists one binding per operation, and tools/list rebuilds >5000 tools from the persisted bindings plus the content-addressed defs blob, never re-parsing the spec. Proven green on cloud and selfhost. * openapi: keep the file-emit hint on the fast serve path The content-addressed serve path rebuilds a tool def from the persisted binding instead of re-parsing the spec. The re-parse path (kept as the fallback) appends the ToolFile emit contract to a file-returning tool's description at the projection step; the fast path bypassed that step and served the bare description, so a file-returning operation lost the contract whenever it was served from the binding (the common case once a spec is persisted). Apply the hint at the same projection step in the fast path, sourced from the binding's response fileHint (already inspected for the output schema), so a file tool carries the contract identically whether served fast or via the fallback. No persist-path or schema change: existing rows pick it up on the next list. Covered by the tool-descriptions e2e scenario. * openapi: stream large specs so the full Microsoft Graph spec fits the Workers isolate Adding the Microsoft Graph provider with the full spec (37MB, ~16,548 operations) 503'd on Cloudflare Workers. The cause is memory, not a bug: a whole-document YAML parse materializes a ~300MB JS tree that exceeds the 128MB-per-isolate limit, and the downstream compile pins that tree alive while it builds every operation, peaking far past the limit. Replace the whole-tree parse on the add/serve paths with a streaming, content-addressed compile that never builds the full tree: - split.ts: structuralSplit() extracts safe byte ranges from the constrained block-YAML profile (top-level keys at col 0, path-items at indent 2, schema entries at indent 4). It parses each path-item and each schema in isolation and discards it, so peak memory stays near a single item (~19MB on workerd) instead of ~300MB. Specs that do not present this profile are rejected with a precise diagnostic rather than silently falling back to the parse that OOMs. - extract.ts: streamOperationBindingsFromStructure() does a 2-pass, same-order compile. Pass 1 derives light schema-free metadata and plans tool paths (global collision resolution); pass 2 builds one binding per operation, flushes in chunks, and drops them. The pass-2 resolver is schema-free, so schema refs stay as lazy $ref strings. - backing.ts: compileAndPersistOpenApiSpecStreaming() streams bindings to storage and writes a content-addressed defs blob keyed by the source hash, shared across every tenant/selection on the same spec. invokeOpenApiBackedTool no longer re-parses the (now full) spec to enrich a present binding; it only re-parses when the binding is entirely absent (a legacy row). Wire the Microsoft Graph add/update path through the streaming primitive with a per-path-item keep filter for the scope selection, and delete the previous reduce-to-a-smaller-document model (and its unit test, superseded by the e2e scenario). Proven by e2e/scenarios/microsoft-graph-full.test.ts, which adds the full catalog and serves >5000 tools, green on both the cloudflare (workerd, the 128MB isolate that previously 503'd) and selfhost targets.

Rhys Sullivan committed 4 months ago

Commit 41137451 parent 108584410 files · +778 −641Browse files

Unified · Split

10 files
  1. e2e/vitest.config.ts +11 −4
  2. packages/plugins/microsoft/src/sdk/graph.test.ts +0 −326
  3. packages/plugins/microsoft/src/sdk/graph.ts +102 −278
  4. packages/plugins/microsoft/src/sdk/index.ts +1 −1
  5. packages/plugins/microsoft/src/sdk/plugin.test.ts +7 −6
  6. packages/plugins/microsoft/src/sdk/plugin.ts +17 −22
  7. packages/plugins/openapi/src/sdk/backing.ts +98 −2
  8. packages/plugins/openapi/src/sdk/extract.ts +140 −1
  9. packages/plugins/openapi/src/sdk/index.ts +15 −1
  10. packages/plugins/openapi/src/sdk/split.ts +387 −0
e2e/vitest.config.ts+11−4
@@ -39,11 +39,18 @@ export default defineConfig({ · 38 unmodified lines
3939 fileParallelism: false,
4040 }),
4141 // The Cloudflare self-host worker (workerd via wrangler dev, dev-auth).
42− // Scoped to the browser-approval scenario for now — the only cross-target
43− // scenario wired for this host; the rest of scenarios/** is not yet
44− // validated against the worker. Shares self-host's single-admin model.
42+ // Scoped to the cross-target scenarios wired for this host; the rest of
43+ // scenarios/** is not yet validated against the worker. The full-graph
44+ // scenario is included on purpose: workerd's 128MB isolate is the exact
45+ // limit the streaming compile + content-addressed serve path defends, so
46+ // it is the one host where that regression must be proven. Shares
47+ // self-host's single-admin model.
4548 project("cloudflare", {
46− include: ["scenarios/browser-approval.test.ts", "cloudflare/**/*.test.ts"],
49+ include: [
50+ "scenarios/browser-approval.test.ts",
51+ "scenarios/microsoft-graph-full.test.ts",
52+ "cloudflare/**/*.test.ts",
53+ ],
4754 fileParallelism: false,
4855 }),
4956 // The Electron desktop app. Only desktop/** scenarios — the desktop
packages/plugins/microsoft/src/sdk/graph.test.ts+0−326
@@ -1,326 +0,0 @@
1−import { describe, expect, it } from "@effect/vitest";
2−import { Effect, Schema } from "effect";
3−
4−import { MICROSOFT_AUTH_TEMPLATE_SLUG, MICROSOFT_GRAPH_CLIENT_CREDENTIALS_SCOPES } from "./presets";
5−import {
6− buildFilteredMicrosoftGraphOpenApiSpec,
7− filterMicrosoftGraphOpenApiSpec,
8− parseMicrosoftGraphDelegatedScopes,
9−} from "./graph";
10−
11−// The filtered preset spec is emitted as JSON, so re-read it with Effect Schema
12−// rather than a YAML parser.
13−const decodeJson = Schema.decodeUnknownSync(Schema.fromJsonString(Schema.Unknown));
14−
15−const graphFixture = `
16−openapi: 3.0.4
17−info:
18− title: Microsoft Graph Fixture
19− version: v1.0
20−servers:
21− - url: https://graph.microsoft.com/v1.0
22−paths:
23− /me:
24− get:
25− operationId: me.GetUser
26− tags:
27− - me.user
28− responses:
29− "200":
30− description: OK
31− /me/messages:
32− get:
33− operationId: me.messages.ListMessages
34− tags:
35− - me.message
36− security:
37− - azureAdDelegated:
38− - Mail.ReadWrite
39− responses:
40− "200":
41− description: OK
42− content:
43− application/json:
44− schema:
45− $ref: "#/components/schemas/messageCollection"
46− /me/onenote/pages:
47− get:
48− operationId: me.onenote.pages.ListPages
49− tags:
50− - me.onenote
51− security:
52− - azureAdDelegated:
53− - Notes.ReadWrite
54− responses:
55− "200":
56− description: OK
57− /teams/{team-id}/channels/{channel-id}/messages:
58− post:
59− operationId: teams.channels.messages.CreateMessage
60− tags:
61− - teams.channel
62− x-ms-permissions:
63− delegated:
64− - ChannelMessage.Send
65− responses:
66− "200":
67− description: OK
68− /sites:
69− get:
70− operationId: sites.ListSites
71− tags:
72− - sites.site
73− responses:
74− "200":
75− description: OK
76−components:
77− schemas:
78− user:
79− type: object
80− messageCollection:
81− type: object
82− properties:
83− value:
84− type: array
85− items:
86− $ref: "#/components/schemas/message"
87− message:
88− type: object
89− properties:
90− id:
91− type: string
92− unusedDirectoryObject:
93− type: object
94−`;
95−
96−describe("Microsoft Graph OpenAPI filtering", () => {
97− it("parses delegated scopes from the generated Microsoft permissions reference", () => {
98− expect(
99− parseMicrosoftGraphDelegatedScopes(`
100−### User.Read
101−
102−| Category | Application | Delegated |
103−|--|--|--|
104−| Identifier | - | e1fe6dd8-ba31-4d61-89e7-88639da4683d |
105−
106−---
107−
108−### AppCatalog.Read.All
109−
110−| Category | Application | Delegated |
111−|--|--|--|
112−| Identifier | e12dae10-5a57-4817-b79d-dfbec5348930 | - |
113−
114−---
115−
116−### Calendars.ReadWrite
117−
118−| Category | Application | Delegated |
119−|--|--|--|
120−| Identifier | ef54d2bf-783f-4e0f-bca1-3210c0444d99 | 1ec239c2-d7c9-4623-a91a-a9775856bb36 |
121−`),
122− ).toEqual(["User.Read", "Calendars.ReadWrite"]);
123− });
124−
125− it.effect("keeps selected paths and injects delegated OAuth", () =>
126− Effect.gen(function* () {
127− const filtered = yield* filterMicrosoftGraphOpenApiSpec(graphFixture, {
128− scopes: ["offline_access", "User.Read", "Mail.ReadWrite"],
129− exactPaths: ["/me"],
130− pathPrefixes: ["/me/messages"],
131− tagPrefixes: [],
132− });
133− const doc = decodeJson(filtered) as {
134− readonly paths: Record<string, unknown>;
135− readonly components: {
136− readonly securitySchemes: Record<string, unknown>;
137− };
138− readonly security: readonly Record<string, readonly string[]>[];
139− };
140−
141− expect(Object.keys(doc.paths).sort()).toEqual(["/me", "/me/messages"]);
142− expect(doc.components.securitySchemes[MICROSOFT_AUTH_TEMPLATE_SLUG]).toBeDefined();
143− expect(doc.security[0]?.[MICROSOFT_AUTH_TEMPLATE_SLUG]).toEqual([
144− "offline_access",
145− "User.Read",
146− "Mail.ReadWrite",
147− ]);
148− }),
149− );
150−
151− it.effect("prunes unreferenced components from filtered specs", () =>
152− Effect.gen(function* () {
153− const filtered = yield* filterMicrosoftGraphOpenApiSpec(graphFixture, {
154− scopes: ["offline_access", "Mail.ReadWrite"],
155− exactPaths: [],
156− pathPrefixes: ["/me/messages"],
157− tagPrefixes: [],
158− });
159− const doc = decodeJson(filtered) as {
160− readonly components: {
161− readonly schemas?: Record<string, unknown>;
162− readonly securitySchemes: Record<string, unknown>;
163− };
164− };
165−
166− expect(Object.keys(doc.components.schemas ?? {}).sort()).toEqual([
167− "message",
168− "messageCollection",
169− ]);
170− expect(doc.components.schemas?.unusedDirectoryObject).toBeUndefined();
171− expect(doc.components.securitySchemes[MICROSOFT_AUTH_TEMPLATE_SLUG]).toBeDefined();
172− }),
173− );
174−
175− it.effect("keeps operations matched by selected Graph scopes", () =>
176− Effect.gen(function* () {
177− const filtered = yield* filterMicrosoftGraphOpenApiSpec(graphFixture, {
178− scopes: ["offline_access", "Notes.ReadWrite", "ChannelMessage.Send"],
179− exactPaths: [],
180− pathPrefixes: [],
181− tagPrefixes: [],
182− });
183− const doc = decodeJson(filtered) as {
184− readonly paths: Record<string, unknown>;
185− };
186−
187− expect(Object.keys(doc.paths).sort()).toEqual([
188− "/me/onenote/pages",
189− "/teams/{team-id}/channels/{channel-id}/messages",
190− ]);
191− }),
192− );
193−
194− it.effect("keeps operations matched by selected Graph tags", () =>
195− Effect.gen(function* () {
196− const filtered = yield* filterMicrosoftGraphOpenApiSpec(graphFixture, {
197− scopes: ["offline_access"],
198− exactPaths: [],
199− pathPrefixes: [],
200− tagPrefixes: ["teams."],
201− });
202− const doc = decodeJson(filtered) as {
203− readonly paths: Record<string, unknown>;
204− };
205−
206− expect(Object.keys(doc.paths)).toEqual(["/teams/{team-id}/channels/{channel-id}/messages"]);
207− }),
208− );
209−
210− it.effect("keeps categorized Graph selections and derives delegated scopes from the spec", () =>
211− Effect.gen(function* () {
212− const filtered = yield* buildFilteredMicrosoftGraphOpenApiSpec(graphFixture, {
213− scopes: ["offline_access"],
214− exactPaths: [],
215− pathPrefixes: ["/me", "/sites", "/teams"],
216− tagPrefixes: [],
217− fullGraphScopes: ["User.Read", "Mail.ReadWrite", "Notes.ReadWrite"],
218− });
219− const doc = decodeJson(filtered.specText) as {
220− readonly paths: Record<string, unknown>;
221− readonly security: readonly Record<string, readonly string[]>[];
222− };
223−
224− expect(Object.keys(doc.paths).sort()).toEqual([
225− "/me",
226− "/me/messages",
227− "/me/onenote/pages",
228− "/sites",
229− "/teams/{team-id}/channels/{channel-id}/messages",
230− ]);
231− expect(filtered.scopes).toEqual([
232− "offline_access",
233− "User.Read",
234− "Mail.ReadWrite",
235− "Notes.ReadWrite",
236− "ChannelMessage.Send",
237− ]);
238− expect(doc.security[0]?.[MICROSOFT_AUTH_TEMPLATE_SLUG]).toEqual(filtered.scopes);
239− }),
240− );
241−
242− it.effect("keeps emulator versioned paths and preserves emulator OAuth endpoints", () =>
243− Effect.gen(function* () {
244− const filtered = yield* filterMicrosoftGraphOpenApiSpec(
245− `
246−openapi: 3.0.3
247−info:
248− title: Microsoft Graph Emulator
249− version: 1.0.0
250−servers:
251− - url: https://microsoft.emulators.dev
252−paths:
253− /v1.0/me:
254− get:
255− operationId: graphUser_GetMyProfile
256− responses:
257− "200":
258− description: OK
259− /v1.0/users:
260− get:
261− operationId: graphUser_List
262− responses:
263− "200":
264− description: OK
265−components:
266− securitySchemes:
267− azureAdDelegated:
268− type: oauth2
269− flows:
270− authorizationCode:
271− authorizationUrl: https://microsoft.emulators.dev/oauth2/v2.0/authorize
272− tokenUrl: https://microsoft.emulators.dev/oauth2/v2.0/token
273− scopes:
274− User.Read: User.Read
275− clientCredentials:
276− tokenUrl: https://microsoft.emulators.dev/oauth2/v2.0/token
277− scopes:
278− https://graph.microsoft.com/.default: https://graph.microsoft.com/.default
279−`,
280− {
281− scopes: ["offline_access", "User.Read", "User.Read.All"],
282− exactPaths: ["/me"],
283− pathPrefixes: ["/users"],
284− tagPrefixes: [],
285− },
286− );
287− const doc = decodeJson(filtered) as {
288− readonly servers: readonly { readonly url: string }[];
289− readonly paths: Record<string, unknown>;
290− readonly components: {
291− readonly securitySchemes: Record<
292− string,
293− {
294− readonly flows: {
295− readonly authorizationCode: {
296− readonly authorizationUrl: string;
297− readonly tokenUrl: string;
298− };
299− readonly clientCredentials: {
300− readonly tokenUrl: string;
301− readonly scopes: Record<string, string>;
302− };
303− };
304− }
305− >;
306− };
307− };
308−
309− expect(doc.servers[0]?.url).toBe("https://microsoft.emulators.dev");
310− expect(Object.keys(doc.paths).sort()).toEqual(["/v1.0/me", "/v1.0/users"]);
311− const flows = doc.components.securitySchemes[MICROSOFT_AUTH_TEMPLATE_SLUG]?.flows;
312− expect(flows?.authorizationCode.authorizationUrl).toBe(
313− "https://microsoft.emulators.dev/oauth2/v2.0/authorize",
314− );
315− expect(flows?.authorizationCode.tokenUrl).toBe(
316− "https://microsoft.emulators.dev/oauth2/v2.0/token",
317− );
318− expect(flows?.clientCredentials.tokenUrl).toBe(
319− "https://microsoft.emulators.dev/oauth2/v2.0/token",
320− );
321− expect(Object.keys(flows?.clientCredentials.scopes ?? {})).toEqual([
322− ...MICROSOFT_GRAPH_CLIENT_CREDENTIALS_SCOPES,
323− ]);
324− }),
325− );
326−});
packages/plugins/microsoft/src/sdk/graph.ts+102−278
@@ -6,17 +6,20 @@ import { AuthTemplateSlug } from "@executor-js/sdk/core"; · 5 unmodified lines
66 import {
77 AuthenticationSchema,
88 OpenApiParseError,
9− parse as parseOpenApiDocument,
9+ parseEntry,
10+ parseHead,
11+ parseSmallComponents,
12+ structuralSplit,
1013 type Authentication,
14+ type KeepPathItem,
1115 type OpenApiIntegrationConfig,
12− type ParsedDocument,
16+ type SpecStructure,
1317 } from "@executor-js/plugin-openapi";
1418
1519 import {
1620 MICROSOFT_AUTHORIZATION_URL,
1721 MICROSOFT_AUTH_TEMPLATE_SLUG,
1822 MICROSOFT_CLIENT_CREDENTIALS_AUTH_TEMPLATE_SLUG,
19− MICROSOFT_GRAPH_BASE_URL,
2023 MICROSOFT_GRAPH_BASE_SCOPES,
2124 MICROSOFT_GRAPH_CLIENT_CREDENTIALS_SCOPES,
2225 MICROSOFT_GRAPH_DELEGATED_DEFAULT_SCOPES,
@@ -43,7 +46,6 @@ export interface MicrosoftGraphSelectionInput { · 20 unmodified lines
4346
4447 export interface MicrosoftGraphSpecBuild {
4548 readonly specText: string;
46− readonly parsedDocument?: ParsedDocument;
4749 readonly specUrl: string;
4850 readonly baseUrl?: string;
4951 readonly authorizationUrl: string;
@@ -93,8 +95,6 @@ const MicrosoftGraphIntegrationConfigSchema = Schema.Struct({ · 43 unmodified lines
9395
9496 const decodeMicrosoftConfig = Schema.decodeUnknownOption(MicrosoftGraphIntegrationConfigSchema);
9597
96−type MicrosoftGraphOpenApiDocument = ParsedDocument & Record<string, unknown>;
97−
9898 export const decodeMicrosoftGraphIntegrationConfig = (
9999 value: unknown,
100100 ): MicrosoftGraphIntegrationConfig | null =>
@@ -174,10 +174,6 @@ const microsoftOAuthTemplate = ( · 73 unmodified lines
174174 const isRecord = (value: unknown): value is Record<string, unknown> =>
175175 value !== null && typeof value === "object" && !Array.isArray(value);
176176
177−// Hoisted so the compiled encoder is built once, not per call. Emits the small
178−// filtered preset spec as JSON, which the OpenAPI SDK's `parse` reads back.
179−const encodeOpenApiDocumentToJson = Schema.encodeSync(Schema.fromJsonString(Schema.Unknown));
180−
181177 const HTTP_METHODS = new Set(["delete", "get", "patch", "post", "put"]);
182178 const BASE_OAUTH_SCOPES = new Set(["offline_access", "openid", "profile", "email"]);
183179
@@ -187,17 +183,6 @@ const firstString = (values: readonly unknown[]): string | undefined => · 3 unmodified lines
187183 const recordValues = (value: unknown): readonly unknown[] =>
188184 isRecord(value) ? Object.values(value) : [];
189185
190−const firstServerUrl = (parsed: Record<string, unknown>): string | undefined => {
191− const servers = parsed.servers;
192− if (!Array.isArray(servers)) return undefined;
193− for (const server of servers) {
194− if (!isRecord(server)) continue;
195− const url = server.url;
196− if (typeof url === "string" && url.trim().length > 0) return url.trim();
197− }
198− return undefined;
199−};
200−
201186 const firstOAuthFlows = (parsed: Record<string, unknown>): readonly Record<string, unknown>[] => {
202187 const components = isRecord(parsed.components) ? parsed.components : {};
203188 const securitySchemes = isRecord(components.securitySchemes) ? components.securitySchemes : {};
@@ -336,25 +321,6 @@ const operationMatchesScope = ( · 132 unmodified lines
336321 (scope) => selectedScopes.has(scope) && !BASE_OAUTH_SCOPES.has(scope),
337322 );
338323
339−const selectedOAuthScopesForPaths = (
340− paths: Record<string, unknown>,
341− requestedScopes: readonly string[],
342− fullGraphScopes: readonly string[] = [],
343−): readonly string[] =>
344− uniqueStrings([
345− ...MICROSOFT_GRAPH_BASE_SCOPES,
346− ...fullGraphScopes,
347− ...requestedScopes.filter((scope) => !BASE_OAUTH_SCOPES.has(scope)),
348− ...Object.values(paths).flatMap((pathItem) => {
349− if (!isRecord(pathItem)) return [];
350− return Object.entries(pathItem).flatMap(([method, operation]) =>
351− HTTP_METHODS.has(method.toLowerCase()) && isRecord(operation)
352− ? permissionScopes(operation, { delegatedOnly: true })
353− : [],
354− );
355− }),
356− ]);
357−
358324 const filterPathItem = (
359325 path: string,
360326 pathItem: Record<string, unknown>,
@@ -390,89 +356,6 @@ const filterPathItem = ( · 29 unmodified lines
390356 return kept;
391357 };
392358
393−const selectMicrosoftGraphPaths = (
394− paths: Record<string, unknown>,
395− options: {
396− readonly scopes: readonly string[];
397− readonly exactPaths: readonly string[];
398− readonly pathPrefixes: readonly string[];
399− readonly tagPrefixes: readonly string[];
400− },
401−): Record<string, unknown> => {
402− const exactPaths = new Set(options.exactPaths);
403− const selectedScopes = new Set(options.scopes);
404− const entries = Object.entries(paths).flatMap(([path, pathItem]) => {
405− if (!isRecord(pathItem)) return [];
406− const filtered = filterPathItem(path, pathItem, {
407− exactPaths,
408− pathPrefixes: options.pathPrefixes,
409− tagPrefixes: options.tagPrefixes,
410− selectedScopes,
411− });
412− return filtered ? ([[path, filtered]] as const) : [];
413− });
414− return Object.fromEntries(entries);
415−};
416−
417−const decodeJsonPointerSegment = (segment: string): string =>
418− decodeURIComponent(segment).replace(/~1/g, "/").replace(/~0/g, "~");
419−
420−const componentRefFromString = (
421− value: string,
422−): { readonly section: string; readonly name: string } | null => {
423− if (!value.startsWith("#/components/")) return null;
424− const [, section, ...nameSegments] = value.slice(2).split("/");
425− if (!section || nameSegments.length === 0) return null;
426− return {
427− section: decodeJsonPointerSegment(section),
428− name: nameSegments.map(decodeJsonPointerSegment).join("/"),
429− };
430−};
431−
432−const collectComponentRefs = (
433− value: unknown,
434−): readonly { readonly section: string; readonly name: string }[] => {
435− if (typeof value === "string") {
436− const ref = componentRefFromString(value);
437− return ref ? [ref] : [];
438− }
439− if (Array.isArray(value)) return value.flatMap(collectComponentRefs);
440− if (!isRecord(value)) return [];
441− return Object.values(value).flatMap(collectComponentRefs);
442−};
443−
444−const prunedComponentsForPaths = (
445− components: Record<string, unknown>,
446− paths: Record<string, unknown>,
447−): Record<string, unknown> => {
448− const pruned: Record<string, Record<string, unknown>> = {};
449− const queue = [...collectComponentRefs(paths)];
450− const seen = new Set<string>();
451−
452− for (let index = 0; index < queue.length; index += 1) {
453− const ref = queue[index];
454− const key = `${ref.section}/${ref.name}`;
455− if (seen.has(key)) continue;
456− seen.add(key);
457−
458− const section = components[ref.section];
459− if (!isRecord(section)) continue;
460− const component = section[ref.name];
461− if (component === undefined) continue;
462−
463− pruned[ref.section] ??= {};
464− pruned[ref.section][ref.name] = component;
465− queue.push(...collectComponentRefs(component));
466− }
467−
468− return pruned;
469−};
470−
471−interface MicrosoftGraphFilterResult {
472− readonly specText: string;
473− readonly scopes: readonly string[];
474−}
475−
476359 export const fetchMicrosoftGraphOpenApiSpec = Effect.fn("Microsoft.fetchGraphOpenApiSpec")(
477360 function* (specUrl: string) {
478361 const client = yield* HttpClient.HttpClient;
@@ -539,128 +422,70 @@ export const fetchMicrosoftGraphPermissionsReference = Effect.fn( · 60 unmodified lines
539422 );
540423 });
541424
542−const parseMicrosoftGraphOpenApiDocument = (
543− specText: string,
544−): Effect.Effect<MicrosoftGraphOpenApiDocument, OpenApiParseError> =>
545− // Delegate to the OpenAPI SDK's lean js-yaml parser. The bespoke eemeli/yaml
546− // parser this replaced built a full CST that peaks ~720MB heap on the 37MB
547− // Graph spec and OOMs the 128MB Cloudflare Workers isolate (measured: HTTP
548− // 503, outcome=exceededMemory). The SDK parser peaks ~315MB and runs in-band
549− // on Workers. `parse` already validates an OpenAPI 3.x object and does not
550− // inline `$ref`s, so peak memory stays bounded for big specs.
551− parseOpenApiDocument(specText).pipe(
552− Effect.mapError(
553− () =>
554− new OpenApiParseError({
555− message: "Failed to parse Microsoft Graph OpenAPI document",
556− }),
557− ),
558− Effect.map((doc) => doc as MicrosoftGraphOpenApiDocument),
425+/**
426+ * Build the per-path-item filter that the streaming compile applies to each
427+ * path-item as it parses the 37MB source. Returns `undefined` for a full-graph
428+ * selection (keep everything). The selection predicate is identical to the old
429+ * two-pass filter: the selected scopes are derived from the PRESET scopes
430+ * (`microsoftGraphScopesForPresetIds`), not the expanded OAuth scopes, so the
431+ * kept operation set matches regardless of caller.
432+ */
433+export const microsoftGraphKeepPathItem = (selection: {
434+ readonly coversFullGraph: boolean;
435+ readonly presetIds: readonly string[];
436+ readonly customScopes: readonly string[];
437+ readonly exactPaths: readonly string[];
438+ readonly pathPrefixes: readonly string[];
439+ readonly tagPrefixes: readonly string[];
440+}): KeepPathItem | undefined => {
441+ if (selection.coversFullGraph) return undefined;
442+ const exactPaths = new Set(selection.exactPaths);
443+ const selectedScopes = new Set(
444+ microsoftGraphScopesForPresetIds(selection.presetIds, selection.customScopes),
559445 );
560−
561−export const buildFilteredMicrosoftGraphOpenApiSpecFromDocument = (
562− parsed: MicrosoftGraphOpenApiDocument,
563− options: {
564− readonly scopes: readonly string[];
565− readonly exactPaths: readonly string[];
566− readonly pathPrefixes: readonly string[];
567− readonly tagPrefixes: readonly string[];
568− readonly baseUrl?: string;
569− readonly authorizationUrl?: string;
570− readonly tokenUrl?: string;
571− readonly clientCredentialsTokenUrl?: string;
572− readonly fullGraphScopes?: readonly string[];
573− },
574−): Effect.Effect<MicrosoftGraphFilterResult, OpenApiParseError> =>
575− Effect.gen(function* () {
576− const paths = parsed.paths;
577− if (!isRecord(paths)) {
578− return yield* new OpenApiParseError({
579− message: "Microsoft Graph OpenAPI document is missing paths",
580− });
581− }
582−
583− const filteredPaths = selectMicrosoftGraphPaths(paths, options);
584− if (Object.keys(filteredPaths).length === 0) {
585− return yield* new OpenApiParseError({
586− message: "Microsoft Graph scope selection did not match any OpenAPI paths",
587− });
588− }
589−
590− const scopes = selectedOAuthScopesForPaths(
591− filteredPaths,
592− options.scopes,
593− options.fullGraphScopes ?? [],
594− );
595− const serverUrl = options.baseUrl ?? firstServerUrl(parsed) ?? MICROSOFT_GRAPH_BASE_URL;
596− const endpoints = resolveOAuthEndpoints(parsed, options);
597− const components = isRecord(parsed.components) ? parsed.components : {};
598− const securitySchemes = isRecord(components.securitySchemes) ? components.securitySchemes : {};
599− const filteredComponents = prunedComponentsForPaths(components, filteredPaths);
600− const next = {
601− ...parsed,
602− info: {
603− ...(isRecord(parsed.info) ? parsed.info : {}),
604− title: "Microsoft Graph",
605− description: "Selected Microsoft Graph workloads from the v1.0 OpenAPI document.",
606− },
607− servers: [{ url: serverUrl }],
608− paths: filteredPaths,
609− components: {
610− ...filteredComponents,
611− securitySchemes: {
612− ...securitySchemes,
613− [MICROSOFT_AUTH_TEMPLATE_SLUG]: {
614− type: "oauth2",
615− flows: {
616− authorizationCode: {
617− authorizationUrl: endpoints.authorizationUrl,
618− tokenUrl: endpoints.tokenUrl,
619− scopes: Object.fromEntries(scopes.map((scope) => [scope, scope])),
620− },
621− clientCredentials: {
622− tokenUrl: endpoints.clientCredentialsTokenUrl,
623− scopes: Object.fromEntries(
624− MICROSOFT_GRAPH_CLIENT_CREDENTIALS_SCOPES.map((scope) => [scope, scope]),
625− ),
626− },
627− },
628− },
629− },
630− },
631− security: [{ [MICROSOFT_AUTH_TEMPLATE_SLUG]: [...scopes] }],
632− };
633−
634− // Serialize as JSON, not YAML. The OpenAPI SDK's `parse` reads JSON (a YAML
635− // subset) on the way back in, so the filtered preset spec round-trips without
636− // pulling in a second YAML library. This is the small filtered spec, never
637− // the 37MB source document.
638− const filteredSpecText = yield* Effect.try({
639− try: () => encodeOpenApiDocumentToJson(next),
640− catch: () =>
641− new OpenApiParseError({
642− message: "Failed to serialize Microsoft Graph OpenAPI document",
643− }),
446+ return (path, pathItem) =>
447+ filterPathItem(path, pathItem, {
448+ exactPaths,
449+ pathPrefixes: selection.pathPrefixes,
450+ tagPrefixes: selection.tagPrefixes,
451+ selectedScopes,
644452 });
645− return { specText: filteredSpecText, scopes };
646− });
647−
648−export const buildFilteredMicrosoftGraphOpenApiSpec = (
649− specText: string,
650− options: Parameters<typeof buildFilteredMicrosoftGraphOpenApiSpecFromDocument>[1],
651−): Effect.Effect<MicrosoftGraphFilterResult, OpenApiParseError> =>
652− Effect.gen(function* () {
653− const parsed = yield* parseMicrosoftGraphOpenApiDocument(specText);
654− return yield* buildFilteredMicrosoftGraphOpenApiSpecFromDocument(parsed, options);
655− });
453+};
656454
657−export const filterMicrosoftGraphOpenApiSpec = (
658− specText: string,
659− options: Parameters<typeof buildFilteredMicrosoftGraphOpenApiSpec>[1],
660−): Effect.Effect<string, OpenApiParseError> =>
661− buildFilteredMicrosoftGraphOpenApiSpec(specText, options).pipe(
662− Effect.map((result) => result.specText),
663− );
455+/**
456+ * Compute the OAuth scopes for the selection by streaming the source path-items
457+ * once (never materializing the whole tree). Mirrors the old
458+ * `selectedOAuthScopesForPaths`: base scopes + full-graph scopes + requested
459+ * scopes + the delegated permission scopes of every kept operation. `keepPathItem`
460+ * (when present) restricts the walk to the filtered operation set, exactly as the
461+ * old code computed scopes over the already-filtered paths.
462+ */
463+const streamSelectedScopes = (
464+ structure: SpecStructure,
465+ requestedScopes: readonly string[],
466+ fullGraphScopes: readonly string[],
467+ keepPathItem?: KeepPathItem,
468+): readonly string[] => {
469+ const collected = [
470+ ...MICROSOFT_GRAPH_BASE_SCOPES,
471+ ...fullGraphScopes,
472+ ...requestedScopes.filter((scope) => !BASE_OAUTH_SCOPES.has(scope)),
473+ ];
474+ for (const range of structure.pathItems) {
475+ const entry = parseEntry(structure.text, range, 2);
476+ if (!entry) continue;
477+ const [path, rawItem] = entry;
478+ if (!isRecord(rawItem)) continue;
479+ const pathItem = keepPathItem ? keepPathItem(path, rawItem) : rawItem;
480+ if (!pathItem) continue;
481+ for (const [method, operation] of Object.entries(pathItem)) {
482+ if (HTTP_METHODS.has(method.toLowerCase()) && isRecord(operation)) {
483+ collected.push(...permissionScopes(operation, { delegatedOnly: true }));
484+ }
485+ }
486+ }
487+ return uniqueStrings(collected);
488+};
664489
665490 export const buildMicrosoftGraphOpenApiSpec = (
666491 input: MicrosoftGraphSelectionInput,
@@ -671,6 +496,24 @@ export const buildMicrosoftGraphOpenApiSpec = ( · 4 unmodified lines
671496 const sourceText = yield* fetchMicrosoftGraphOpenApiSpec(selection.specUrl).pipe(
672497 Effect.provide(httpClientLayer),
673498 );
499+
500+ // Structural split is the only entry point: parsing the whole 37MB tree
501+ // OOMs the 128MB Workers isolate (measured: HTTP 503). No fallback. A spec
502+ // outside the streamable block-YAML profile is a hard error on this path;
503+ // arbitrary user specs still go through the generic openapi plugin.
504+ const structure = structuralSplit(sourceText);
505+ if (!structure) {
506+ return yield* new OpenApiParseError({
507+ message:
508+ "Microsoft Graph OpenAPI document is not in the streamable block-YAML profile; cannot compile it in-band on Workers.",
509+ });
510+ }
511+
512+ // Head + small components (servers + securitySchemes) parse cheaply and
513+ // carry everything `resolveOAuthEndpoints` needs.
514+ const headDoc = { ...parseHead(structure), components: parseSmallComponents(structure) };
515+ const endpoints = resolveOAuthEndpoints(headDoc, selection);
516+
674517 const permissionsReference =
675518 selection.coversFullGraph === true
676519 ? yield* fetchMicrosoftGraphPermissionsReference().pipe(Effect.provide(httpClientLayer))
@@ -678,46 +521,27 @@ export const buildMicrosoftGraphOpenApiSpec = ( · 1 unmodified line
678521 const fullGraphScopes = permissionsReference
679522 ? parseMicrosoftGraphDelegatedScopes(permissionsReference)
680523 : [];
681− const parsed = yield* parseMicrosoftGraphOpenApiDocument(sourceText);
682− const endpoints = resolveOAuthEndpoints(parsed, selection);
683− const graphPaths = parsed.paths;
684− if (!isRecord(graphPaths)) {
685− return yield* new OpenApiParseError({
686− message: "Microsoft Graph OpenAPI document is missing paths",
687− });
688− }
689− if (selection.coversFullGraph === true) {
690− const scopes =
691− selection.customScopes.length === 0
692− ? MICROSOFT_GRAPH_DELEGATED_DEFAULT_SCOPES
693− : selectedOAuthScopesForPaths(
694− graphPaths,
695− uniqueStrings([...MICROSOFT_GRAPH_BASE_SCOPES, ...selection.customScopes]),
696− fullGraphScopes,
697− );
698− return {
699− ...selection,
700− specText: sourceText,
701− parsedDocument: parsed,
702− scopes,
703− authorizationUrl: endpoints.authorizationUrl,
704− tokenUrl: endpoints.tokenUrl,
705− clientCredentialsTokenUrl: endpoints.clientCredentialsTokenUrl,
706− authenticationTemplate: microsoftOAuthTemplate(scopes, endpoints),
707− };
708− }
709− const filtered = yield* buildFilteredMicrosoftGraphOpenApiSpecFromDocument(parsed, {
710− ...selection,
711− scopes: selection.scopes,
712− fullGraphScopes,
713− });
524+
525+ const keepPathItem = microsoftGraphKeepPathItem(selection);
526+ const scopes =
527+ selection.coversFullGraph === true && selection.customScopes.length === 0
528+ ? [...MICROSOFT_GRAPH_DELEGATED_DEFAULT_SCOPES]
529+ : streamSelectedScopes(
530+ structure,
531+ selection.coversFullGraph === true
532+ ? uniqueStrings([...MICROSOFT_GRAPH_BASE_SCOPES, ...selection.customScopes])
533+ : selection.scopes,
534+ fullGraphScopes,
535+ keepPathItem,
536+ );
537+
714538 return {
715539 ...selection,
716− specText: filtered.specText,
717− scopes: filtered.scopes,
540+ specText: sourceText,
541+ scopes,
718542 authorizationUrl: endpoints.authorizationUrl,
719543 tokenUrl: endpoints.tokenUrl,
720544 clientCredentialsTokenUrl: endpoints.clientCredentialsTokenUrl,
721− authenticationTemplate: microsoftOAuthTemplate(filtered.scopes, endpoints),
545+ authenticationTemplate: microsoftOAuthTemplate(scopes, endpoints),
722546 };
723547 });
packages/plugins/microsoft/src/sdk/index.ts+1−1
@@ -30,7 +30,7 @@ export { · 29 unmodified lines
3030 decodeMicrosoftGraphIntegrationConfig,
3131 fetchMicrosoftGraphOpenApiSpec,
3232 fetchMicrosoftGraphPermissionsReference,
33− filterMicrosoftGraphOpenApiSpec,
33+ microsoftGraphKeepPathItem,
3434 parseMicrosoftGraphDelegatedScopes,
3535 type MicrosoftGraphIntegrationConfig,
3636 type MicrosoftGraphSelectionInput,
packages/plugins/microsoft/src/sdk/plugin.test.ts+7−6
@@ -343,12 +343,13 @@ describe("Microsoft Graph provider", () => { · 342 unmodified lines
343343 ...MICROSOFT_GRAPH_DELEGATED_DEFAULT_SCOPES,
344344 ]);
345345
346− // Full-graph add routes through the streaming `parsedDocument` persist
347− // branch (the path the real 37MB spec takes): it persists each op's
348− // binding plus a `description` and writes the content-addressed defs
349− // blob, never re-parsing on serve. Read the operations back through the
350− // live serve path to prove they landed in storage AND that the serve
351− // fast path rebuilds tools from the persisted bindings (no spec parse).
346+ // Full-graph add routes through the streaming compile (the path the
347+ // real 37MB spec takes): the source text is structurally split and each
348+ // op's binding plus a `description` is persisted, alongside the
349+ // content-addressed defs blob, never materializing the whole-document
350+ // tree. Read the operations back through the live serve path to prove
351+ // they landed in storage AND that the serve fast path rebuilds tools
352+ // from the persisted bindings (no spec parse).
352353 yield* executor.connections.create({
353354 owner: "org",
354355 name: ConnectionName.make("full"),
packages/plugins/microsoft/src/sdk/plugin.ts+17−22
@@ -19,27 +19,26 @@ import { · 18 unmodified lines
1919 } from "@executor-js/sdk/core";
2020 import { describeApiKeyAuthMethod } from "@executor-js/sdk/http-auth";
2121 import {
22− compileAndPersistOpenApiOperations,
23− compileAndPersistOpenApiSpec,
22+ compileAndPersistOpenApiSpecStreaming,
2423 decodeOpenApiIntegrationConfig,
2524 invokeOpenApiBackedTool,
2625 makeDefaultOpenapiStore,
2726 normalizeOpenApiAuthInputs,
2827 OpenApiExtractionError,
29− OpenApiParseError,
3028 resolveOpenApiBackedAnnotations,
3129 resolveOpenApiBackedTools,
3230 type Authentication,
3331 type AuthenticationInput,
3432 type OpenApiPersistResult,
3533 type OpenapiStore,
36− type ParsedDocument,
3734 } from "@executor-js/plugin-openapi";
3835
3936 import {
4037 buildMicrosoftGraphOpenApiSpec,
4138 decodeMicrosoftGraphIntegrationConfig,
39+ microsoftGraphKeepPathItem,
4240 type MicrosoftGraphIntegrationConfig,
41+ type MicrosoftGraphSpecBuild,
4342 } from "./graph";
4443 import {
4544 MICROSOFT_CLIENT_CREDENTIALS_AUTH_TEMPLATE_SLUG,
@@ -127,26 +126,22 @@ const makeMicrosoftPluginExtension = ( · 81 unmodified lines
127126 httpClientLayer: Layer.Layer<HttpClient.HttpClient, never, never>,
128127 ) => {
129128 const persistGraphOperations = (
130− graph: { readonly parsedDocument?: ParsedDocument; readonly specText: string },
129+ graph: MicrosoftGraphSpecBuild,
131130 integration: string,
132131 specHash: string,
133− ): Effect.Effect<
134− OpenApiPersistResult,
135− OpenApiExtractionError | OpenApiParseError | StorageFailure
136− > =>
137− graph.parsedDocument !== undefined
138− ? compileAndPersistOpenApiOperations({
139− doc: graph.parsedDocument,
140− integration,
141− storage: ctx.storage,
142− specHash,
143− })
144− : compileAndPersistOpenApiSpec({
145− specText: graph.specText,
146− integration,
147− storage: ctx.storage,
148− specHash,
149− });
132+ ): Effect.Effect<OpenApiPersistResult, OpenApiExtractionError | StorageFailure> =>
133+ // Stream the (full 37MB) source straight to persisted bindings + a
134+ // content-addressed defs blob, never materializing the whole-document tree
135+ // that OOMs the 128MB Workers isolate. `keepPathItem` applies the Microsoft
136+ // Graph scope selection per path-item during the stream; a full-graph
137+ // selection returns `undefined` (keep everything).
138+ compileAndPersistOpenApiSpecStreaming({
139+ specText: graph.specText,
140+ integration,
141+ storage: ctx.storage,
142+ specHash,
143+ keepPathItem: microsoftGraphKeepPathItem(graph),
144+ });
150145
151146 const addGraph = (config: MicrosoftGraphConfig) =>
152147 Effect.gen(function* () {
packages/plugins/openapi/src/sdk/backing.ts+98−2
@@ -21,10 +21,16 @@ import { · 20 unmodified lines
2121 type OpenApiIntegrationConfig,
2222 } from "./config";
2323 import { OpenApiExtractionError, OpenApiParseError } from "./errors";
24−import { buildInputSchema, extract, streamOperationBindings } from "./extract";
24+import {
25+ buildInputSchema,
26+ extract,
27+ streamOperationBindings,
28+ streamOperationBindingsFromStructure,
29+} from "./extract";
2530 import { compileToolDefinitions, type ToolDefinition } from "./definitions";
2631 import { annotationsForOperation, invokeWithLayer } from "./invoke";
2732 import { parse, type ParsedDocument } from "./parse";
33+import { parseEntry, structuralSplit, type KeepPathItem, type SpecStructure } from "./split";
2834 import { type OpenapiStore, type StoredOperation } from "./store";
2935 import { OperationBinding } from "./types";
3036
@@ -282,6 +288,32 @@ export const buildDefsJson = (doc: ParsedDocument): string => { · 251 unmodified lines
282288 return `${json}}`;
283289 };
284290
291+/**
292+ * Streaming twin of `buildDefsJson`: serialize the content-addressed defs blob
293+ * from a `SpecStructure` by parsing each `components.schemas` entry in isolation
294+ * (indent-4 range) rather than from a whole-document parse. Used by the fully
295+ * streaming add path so the 37MB Microsoft Graph spec never builds its
296+ * ~300MB tree. The blob carries *all* source schemas (it is shared across every
297+ * tenant/selection on the same spec hash); extra unreferenced `$defs` are
298+ * harmless. Like `buildDefsJson`, normalizing + stringifying per entry keeps the
299+ * whole normalized tree from being co-resident with any parsed schema, and the
300+ * ConsString accumulation avoids the join-doubling of an array build.
301+ */
302+export const buildDefsJsonStreaming = (structure: SpecStructure): string => {
303+ let json = "{";
304+ let first = true;
305+ for (const range of structure.schemas) {
306+ const entry = parseEntry(structure.text, range, 4);
307+ if (!entry) continue;
308+ const [name, schema] = entry;
309+ const serialized = JSON.stringify(normalizeOpenApiRefs(schema));
310+ if (serialized === undefined) continue;
311+ json += `${first ? "" : ","}${JSON.stringify(name)}:${serialized}`;
312+ first = false;
313+ }
314+ return `${json}}`;
315+};
316+
285317 const DefsJson = Schema.Record(Schema.String, Schema.Unknown);
286318 /** Decode the content-addressed defs blob back into the shared `definitions`
287319 * map. Returns `None` on a corrupt/non-object blob so the serve path falls
@@ -402,6 +434,65 @@ export const compileAndPersistOpenApiSpec = ({ · 114 unmodified lines
402434 });
403435 });
404436
437+/**
438+ * Fully streaming add/update path: compile + persist operation bindings (and the
439+ * content-addressed defs blob) straight from spec *text*, without ever parsing
440+ * the whole document. The text is structurally split, then each path-item and
441+ * each schema is parsed in isolation and discarded, so peak memory stays near
442+ * one item rather than the ~300MB whole-tree parse that OOMs a 128MB Workers
443+ * isolate on the 37MB Microsoft Graph spec.
444+ *
445+ * There is deliberately no whole-parse fallback: a spec that does not present
446+ * the streamable block-YAML profile (no top-level `paths:` block) is a hard
447+ * `OpenApiExtractionError`, because the fallback is exactly the OOM this path
448+ * exists to avoid. `keepPathItem` optionally filters/trims path-items (the
449+ * Microsoft Graph scope selection), so the same primitive serves a full-spec
450+ * compile and a selection identically.
451+ */
452+export const compileAndPersistOpenApiSpecStreaming = ({
453+ specText,
454+ integration,
455+ storage,
456+ specHash,
457+ chunkSize,
458+ keepPathItem,
459+}: {
460+ readonly specText: string;
461+ readonly integration: string;
462+ readonly storage: OpenapiStore;
463+ readonly specHash?: string;
464+ readonly chunkSize?: number;
465+ readonly keepPathItem?: KeepPathItem;
466+}): Effect.Effect<OpenApiPersistResult, OpenApiExtractionError | StorageFailure> =>
467+ Effect.gen(function* () {
468+ const structure = structuralSplit(specText);
469+ if (!structure) {
470+ return yield* new OpenApiExtractionError({
471+ message:
472+ "OpenAPI spec is not in the streamable block-YAML profile (no top-level `paths:` block); cannot stream-compile a spec this large in-band.",
473+ });
474+ }
475+ yield* storage.removeOperations(integration);
476+ const result = yield* streamOperationBindingsFromStructure(
477+ structure,
478+ { chunkSize: chunkSize ?? 500, keepPathItem },
479+ (chunk) =>
480+ storage.appendOperations(
481+ integration,
482+ chunk.map((item) => ({
483+ integration,
484+ toolName: item.toolName,
485+ binding: item.binding,
486+ description: item.description,
487+ })),
488+ ),
489+ );
490+ if (specHash != null) {
491+ yield* storage.putDefs(specHash, buildDefsJsonStreaming(structure));
492+ }
493+ return result;
494+ });
495+
405496 export const loadOpenApiSpecText = (
406497 storage: OpenapiStore,
407498 config: OpenApiIntegrationConfig,
@@ -464,7 +555,12 @@ export const invokeOpenApiBackedTool = (input: { · 56 unmodified lines
464555 const config = decodeOpenApiIntegrationConfig(input.credential.config);
465556
466557 let binding = (yield* input.ctx.storage.getOperation(integration, input.toolRow.name))?.binding;
467− if ((!binding || Option.isNone(binding.responseBody)) && config) {
558+ // Only re-parse when the binding is entirely absent (a legacy row predating
559+ // persisted bindings). A present binding is authoritative even if it has no
560+ // responseBody: the persisted spec is now the *full* source (37MB for
561+ // Microsoft Graph), so re-parsing it here to "enrich" a binding would OOM
562+ // the isolate. A genuinely body-less operation must serve from its binding.
563+ if (!binding && config) {
468564 const specText = yield* loadOpenApiSpecText(input.ctx.storage, config).pipe(
469565 Effect.catch(() => Effect.succeed(null)),
470566 );
packages/plugins/openapi/src/sdk/extract.ts+140−1
@@ -1,8 +1,16 @@
11 import { Effect, Option } from "effect";
22
3−import { planToolPaths, type OperationPathInput } from "./definitions";
3+import { planToolPaths, type OperationPathInput, type PlannedToolPath } from "./definitions";
44 import { OpenApiExtractionError } from "./errors";
55 import type { ParsedDocument } from "./parse";
6+import {
7+ parseEntry,
8+ parseHead,
9+ parseSmallComponents,
10+ type ByteRange,
11+ type KeepPathItem,
12+ type SpecStructure,
13+} from "./split";
614 import {
715 declaredContents,
816 DocResolver,
@@ -622,3 +630,134 @@ export const streamOperationBindings = <E, R>( · 613 unmodified lines
622630
623631 return { toolCount: plans.length, toolNames: plans.map((plan) => plan.toolPath) };
624632 }).pipe(Effect.withSpan("OpenApi.streamOperationBindings"));
633+
634+const isPathItemValue = (value: unknown): value is Record<string, unknown> =>
635+ value !== null && typeof value === "object" && !Array.isArray(value);
636+
637+/**
638+ * Stream invocation bindings straight from a `SpecStructure` (the structural
639+ * split of a large spec) without ever materializing the whole-document tree.
640+ *
641+ * This is the fully-streaming compile path: it never parses the spec whole.
642+ * Each path-item is parsed in isolation twice (pass 1 for light tool-path
643+ * planning metadata, pass 2 to build the heavy binding) and discarded, so peak
644+ * memory stays near the size of a single path-item plus the raw text, even for
645+ * the 37MB / 16.5k-operation Microsoft Graph spec that OOMs a whole-tree parse.
646+ * Re-parsing in pass 2 (rather than holding pass-1 tree references) is the
647+ * deliberate CPU-for-memory trade that keeps peak at one-path-item level.
648+ *
649+ * `keepPathItem`, when given, filters (and may trim) each path-item, so the same
650+ * primitive serves both a full-spec compile (no filter) and a selection (e.g.
651+ * the Microsoft Graph scope filter) with identical streaming guarantees. It is
652+ * applied identically in both passes so the per-operation index stays aligned.
653+ *
654+ * Schemas are never resolved here: parameter / requestBody / response component
655+ * `$ref`s resolve against the small (schema-free) components built from the
656+ * structure; `#/components/schemas/X` refs stay as strings in the binding and
657+ * are normalized + served from the content-addressed defs blob. Returns the
658+ * resolved tool names in sorted order, matching `compileToolDefinitions`.
659+ */
660+export const streamOperationBindingsFromStructure = <E, R>(
661+ structure: SpecStructure,
662+ options: { readonly chunkSize: number; readonly keepPathItem?: KeepPathItem },
663+ onChunk: (chunk: readonly OperationBindingChunk[]) => Effect.Effect<void, E, R>,
664+): Effect.Effect<{ readonly toolCount: number; readonly toolNames: readonly string[] }, E, R> =>
665+ Effect.gen(function* () {
666+ const { chunkSize, keepPathItem } = options;
667+
668+ // Parse one path-item range to its kept (optionally trimmed) value, applying
669+ // `keepPathItem` identically in both passes so the operation index aligns.
670+ const keptPathItem = (range: ByteRange): readonly [string, PathItemObject] | null => {
671+ const entry = parseEntry(structure.text, range, 2);
672+ if (!entry) return null;
673+ const [path, rawValue] = entry;
674+ if (!isPathItemValue(rawValue)) return null;
675+ if (!keepPathItem) return [path, rawValue as PathItemObject];
676+ const kept = keepPathItem(path, rawValue);
677+ return kept ? [path, kept as PathItemObject] : null;
678+ };
679+
680+ // Pass 1 (light): collect schema-free tool-path planning metadata in
681+ // document order. No bindings, no schemas; one path-item resident at a time.
682+ const inputs: OperationPathInput[] = [];
683+ for (const range of structure.pathItems) {
684+ const kept = keptPathItem(range);
685+ if (!kept) continue;
686+ const [path, pathItem] = kept;
687+ for (const method of HTTP_METHODS) {
688+ const operation = pathItem[method];
689+ if (!operation) continue;
690+ const resolvedPathTemplate = explicitPathTemplate(operation) ?? path;
691+ const tags = (operation.tags ?? []).filter((t) => t.trim().length > 0);
692+ inputs.push({
693+ operationId: deriveOperationId(method, path, operation),
694+ explicitToolPath: explicitToolPath(operation),
695+ method,
696+ pathTemplate: resolvedPathTemplate,
697+ tag0: tags[0],
698+ });
699+ }
700+ }
701+
702+ // Global, schema-free collision resolution + sort. `plan.operationIndex`
703+ // indexes back into `inputs` (document order), so a flat array recovers each
704+ // operation's assigned tool path during the document-order pass 2.
705+ const plans = planToolPaths(inputs);
706+ const planByOpIndex: (PlannedToolPath | undefined)[] = new Array(inputs.length);
707+ for (const plan of plans) planByOpIndex[plan.operationIndex] = plan;
708+
709+ // Pass 2 (heavy, streamed): re-parse each path-item in the same document
710+ // order, build a binding per operation, flush a chunk once it fills, then
711+ // drop it. The resolver is schema-free (small components only); schema refs
712+ // stay as `$ref` strings in the bindings.
713+ // oxlint-disable-next-line executor/no-double-cast -- boundary: parseHead/parseSmallComponents return Record<string, unknown>, which does not structurally match the OpenAPIV3 Document union; the schema-free resolver doc (head + small components, empty paths) is only read for .servers (extractServers) and .components ($ref resolution).
714+ const resolverDoc = {
715+ ...parseHead(structure),
716+ paths: {},
717+ components: parseSmallComponents(structure),
718+ } as unknown as ParsedDocument;
719+ const r = new DocResolver(resolverDoc);
720+ const docServers = extractServers(resolverDoc);
721+
722+ let opIndex = 0;
723+ let chunk: OperationBindingChunk[] = [];
724+ for (const range of structure.pathItems) {
725+ const kept = keptPathItem(range);
726+ if (!kept) continue;
727+ const [path, pathItem] = kept;
728+ for (const method of HTTP_METHODS) {
729+ const operation = pathItem[method];
730+ if (!operation) continue;
731+ const plan = planByOpIndex[opIndex];
732+ opIndex += 1;
733+ if (!plan) continue;
734+ const resolvedPathTemplate = explicitPathTemplate(operation) ?? path;
735+ const parameters = extractParameters(pathItem, operation, r);
736+ const requestBody = extractRequestBody(operation, r);
737+ const responseBody = extractResponseBody(operation, r);
738+ const servers = operationServers(pathItem, operation, docServers);
739+ chunk.push({
740+ toolName: plan.toolPath,
741+ description:
742+ operation.description ??
743+ operation.summary ??
744+ `${method.toUpperCase()} ${resolvedPathTemplate}`,
745+ binding: OperationBinding.make({
746+ method,
747+ servers,
748+ pathTemplate: resolvedPathTemplate,
749+ parameters,
750+ requestBody: Option.fromNullishOr(requestBody),
751+ responseBody: Option.fromNullishOr(responseBody),
752+ }),
753+ });
754+ if (chunk.length >= chunkSize) {
755+ yield* onChunk(chunk);
756+ chunk = [];
757+ }
758+ }
759+ }
760+ if (chunk.length > 0) yield* onChunk(chunk);
761+
762+ return { toolCount: plans.length, toolNames: plans.map((plan) => plan.toolPath) };
763+ }).pipe(Effect.withSpan("OpenApi.streamOperationBindingsFromStructure"));
packages/plugins/openapi/src/sdk/index.ts+15−1
@@ -1,9 +1,23 @@
11 export { parse, resolveSpecText, fetchSpecText } from "./parse";
2−export { extract } from "./extract";
2+export { extract, streamOperationBindingsFromStructure } from "./extract";
3+export {
4+ structuralSplit,
5+ isStreamableSpec,
6+ indexSchemas,
7+ collectReferencedSchemas,
8+ parseEntry,
9+ parseHead,
10+ parseSmallComponents,
11+ type SpecStructure,
12+ type ByteRange,
13+ type KeepPathItem,
14+} from "./split";
315 export { invoke, invokeWithLayer, annotationsForOperation } from "./invoke";
416 export {
17+ buildDefsJsonStreaming,
518 compileAndPersistOpenApiOperations,
619 compileAndPersistOpenApiSpec,
20+ compileAndPersistOpenApiSpecStreaming,
721 compileOpenApiDocument,
822 compileOpenApiSpec,
923 extractOpenApiUpstreamMessage,
packages/plugins/openapi/src/sdk/split.ts+387−0
@@ -0,0 +1,387 @@
1+/**
2+ * Structural splitter for large OpenAPI documents written in clean block YAML.
3+ *
4+ * The whole-document parse of a 37MB spec (Microsoft Graph: 16.5k operations,
5+ * 8.2k schemas) builds a ~300MB JS tree that OOMs the 128MB Cloudflare Workers
6+ * isolate. This splitter avoids ever holding that tree: it scans the text once,
7+ * recording the byte range of each top-level key, each path-item (the indent-2
8+ * entries under `paths:`), and each schema (the indent-4 entries under
9+ * `components.schemas:`). The streaming compile then slices one item at a time,
10+ * de-indents it back to column 0, hands the isolated fragment to a real YAML
11+ * parser, and discards the result before moving on. Peak memory stays near the
12+ * size of the largest single item plus the raw text, not the parsed whole.
13+ *
14+ * This is not a YAML reimplementation: it extracts safe byte ranges from a
15+ * constrained document shape, then defers every actual parse to `js-yaml`. It
16+ * is only valid for the block-YAML profile `isStreamableSpec` accepts (2-space
17+ * block maps, top-level keys at column 0, no anchors/aliases/merge keys). Block
18+ * scalars (`|` / `>`) in descriptions are tracked so their indented content is
19+ * never mistaken for structure.
20+ */
21+
22+import { JSON_SCHEMA, load as parseYamlDocument } from "js-yaml";
23+
24+export interface ByteRange {
25+ readonly start: number;
26+ readonly end: number;
27+}
28+
29+export interface SpecStructure {
30+ /** Raw spec text. Every range below indexes into this string. */
31+ readonly text: string;
32+ /** Byte ranges of the top-level keys that are not `paths` / `components`
33+ * (openapi, info, servers, tags, security, ...). Concatenated and parsed as
34+ * one small document for the head (servers, info). */
35+ readonly headRanges: readonly ByteRange[];
36+ /** One range per path-item: the indent-2 entries under `paths:`. */
37+ readonly pathItems: readonly ByteRange[];
38+ /** One range per schema entry: the indent-4 entries under
39+ * `components.schemas:`. */
40+ readonly schemas: readonly ByteRange[];
41+ /** Ranges of the indent-2 `components` subkeys we keep whole because they are
42+ * small and may be `$ref`'d by a kept operation (parameters / requestBodies /
43+ * responses / headers / links / securitySchemes). Excludes the huge `schemas`
44+ * (streamed and pruned separately) and `examples` (never referenced by a
45+ * binding). */
46+ readonly smallComponentRanges: readonly ByteRange[];
47+}
48+
49+const SPACE = 32;
50+const HASH = 35;
51+const DASH = 45;
52+
53+/** `components` subkeys kept whole during a reduce: small, and a kept operation
54+ * may `$ref` into any of them. `schemas` is streamed/pruned separately and
55+ * `examples` is dropped (large, never referenced by a binding). */
56+const SMALL_COMPONENT_SECTIONS = new Set([
57+ "parameters",
58+ "requestBodies",
59+ "responses",
60+ "headers",
61+ "links",
62+ "securitySchemes",
63+]);
64+
65+/** Count of leading spaces on the line starting at `lineStart`. */
66+const indentOf = (text: string, lineStart: number, lineEnd: number): number => {
67+ let i = lineStart;
68+ while (i < lineEnd && text.charCodeAt(i) === SPACE) i++;
69+ return i - lineStart;
70+};
71+
72+/** True for a blank (whitespace-only) or comment line. */
73+const isBlankOrComment = (text: string, contentStart: number, lineEnd: number): boolean =>
74+ contentStart >= lineEnd || text.charCodeAt(contentStart) === HASH;
75+
76+/**
77+ * True when the line opens a block scalar (`key: |`, `key: >-`, etc.), meaning
78+ * every following more-indented line is literal content, not structure.
79+ */
80+const opensBlockScalar = (line: string): boolean =>
81+ /:\s*[|>][+-]?\d*\s*(#.*)?$/.test(line) || /^\s*[|>][+-]?\d*\s*(#.*)?$/.test(line);
82+
83+interface LineCursor {
84+ readonly lineStart: number;
85+ readonly lineEnd: number;
86+ readonly nextStart: number;
87+ readonly indent: number;
88+ readonly contentStart: number;
89+}
90+
91+/** Advance to the next significant line at offset `from`, returning null at EOF.
92+ * Blank lines, comments, and block-scalar content (lines indented deeper than
93+ * the scalar key, tracked by the caller) are still returned; the caller skips
94+ * them. */
95+const lineAt = (text: string, from: number, limit: number): LineCursor | null => {
96+ if (from >= limit) return null;
97+ let lineEnd = text.indexOf("\n", from);
98+ if (lineEnd === -1 || lineEnd > limit) lineEnd = limit;
99+ const indent = indentOf(text, from, lineEnd);
100+ const contentStart = from + indent;
101+ return {
102+ lineStart: from,
103+ lineEnd,
104+ nextStart: lineEnd + 1,
105+ indent,
106+ contentStart,
107+ };
108+};
109+
110+/**
111+ * Find the start offset of every key line at exactly `indent` within
112+ * `[start, end)`, skipping blank/comment lines, deeper child lines, and
113+ * block-scalar content. Sequence items (`- `) at the target indent are not
114+ * treated as keys. Returns offsets in document order.
115+ */
116+const keyLineStartsAtIndent = (
117+ text: string,
118+ start: number,
119+ end: number,
120+ indent: number,
121+): number[] => {
122+ const starts: number[] = [];
123+ let blockScalarIndent = -1;
124+ let pos = start;
125+ while (pos < end) {
126+ const line = lineAt(text, pos, end);
127+ if (!line) break;
128+ pos = line.nextStart;
129+ if (isBlankOrComment(text, line.contentStart, line.lineEnd)) continue;
130+ if (blockScalarIndent >= 0) {
131+ if (line.indent > blockScalarIndent) continue;
132+ blockScalarIndent = -1;
133+ }
134+ if (line.indent === indent && text.charCodeAt(line.contentStart) !== DASH) {
135+ starts.push(line.lineStart);
136+ }
137+ if (line.indent <= indent && opensBlockScalar(text.slice(line.contentStart, line.lineEnd))) {
138+ blockScalarIndent = line.indent;
139+ }
140+ }
141+ return starts;
142+};
143+
144+/** Contiguous ranges from a sorted list of block-start offsets within
145+ * `[blockStart, blockEnd)`: each range runs to the next start (or blockEnd). */
146+const rangesFromStarts = (starts: readonly number[], blockEnd: number): ByteRange[] =>
147+ starts.map((s, i) => ({ start: s, end: i + 1 < starts.length ? starts[i + 1]! : blockEnd }));
148+
149+/** The simple (unquoted) key name on a key line, e.g. `paths`, `schemas`. */
150+const keyNameAt = (text: string, lineStart: number, lineEnd: number): string => {
151+ const indent = indentOf(text, lineStart, lineEnd);
152+ const colon = text.indexOf(":", lineStart + indent);
153+ const keyEnd = colon === -1 || colon > lineEnd ? lineEnd : colon;
154+ return text.slice(lineStart + indent, keyEnd).trim();
155+};
156+
157+/**
158+ * Scan a document into its structural ranges. Pure and synchronous; never
159+ * parses. Returns null when the document does not present the expected shape
160+ * (no `paths:` block), so the caller can fall back to a whole-document parse.
161+ */
162+export const structuralSplit = (text: string): SpecStructure | null => {
163+ const len = text.length;
164+
165+ // Top-level keys (column 0). Record each key's start + name, respecting
166+ // block scalars whose content could sit at column >= 1.
167+ const topStarts: number[] = [];
168+ const topNames: string[] = [];
169+ {
170+ let blockScalarIndent = -1;
171+ let pos = 0;
172+ while (pos < len) {
173+ const line = lineAt(text, pos, len);
174+ if (!line) break;
175+ pos = line.nextStart;
176+ if (isBlankOrComment(text, line.contentStart, line.lineEnd)) continue;
177+ if (blockScalarIndent >= 0) {
178+ if (line.indent > blockScalarIndent) continue;
179+ blockScalarIndent = -1;
180+ }
181+ if (line.indent === 0) {
182+ topStarts.push(line.lineStart);
183+ topNames.push(keyNameAt(text, line.lineStart, line.lineEnd));
184+ }
185+ if (opensBlockScalar(text.slice(line.contentStart, line.lineEnd))) {
186+ blockScalarIndent = line.indent;
187+ }
188+ }
189+ }
190+
191+ const topRanges = rangesFromStarts(topStarts, len);
192+ const pathsIdx = topNames.indexOf("paths");
193+ if (pathsIdx === -1) return null;
194+ const componentsIdx = topNames.indexOf("components");
195+
196+ const headRanges: ByteRange[] = [];
197+ for (let i = 0; i < topRanges.length; i++) {
198+ if (i === pathsIdx || i === componentsIdx) continue;
199+ headRanges.push(topRanges[i]!);
200+ }
201+
202+ // Path-items: indent-2 keys inside the `paths:` block (after its key line).
203+ const pathsRange = topRanges[pathsIdx]!;
204+ const pathsBodyStart =
205+ lineAt(text, pathsRange.start, pathsRange.end)?.nextStart ?? pathsRange.end;
206+ const pathItems = rangesFromStarts(
207+ keyLineStartsAtIndent(text, pathsBodyStart, pathsRange.end, 2),
208+ pathsRange.end,
209+ );
210+
211+ const schemas: ByteRange[] = [];
212+ const smallComponentRanges: ByteRange[] = [];
213+ if (componentsIdx !== -1) {
214+ const componentsRange = topRanges[componentsIdx]!;
215+ const componentsBodyStart =
216+ lineAt(text, componentsRange.start, componentsRange.end)?.nextStart ?? componentsRange.end;
217+ const subStarts = keyLineStartsAtIndent(text, componentsBodyStart, componentsRange.end, 2);
218+ const subRanges = rangesFromStarts(subStarts, componentsRange.end);
219+ for (const range of subRanges) {
220+ const name = keyNameAt(text, range.start, lineAt(text, range.start, range.end)!.lineEnd);
221+ if (name === "schemas") {
222+ const bodyStart = lineAt(text, range.start, range.end)?.nextStart ?? range.end;
223+ for (const s of rangesFromStarts(
224+ keyLineStartsAtIndent(text, bodyStart, range.end, 4),
225+ range.end,
226+ )) {
227+ schemas.push(s);
228+ }
229+ } else if (SMALL_COMPONENT_SECTIONS.has(name)) {
230+ smallComponentRanges.push(range);
231+ }
232+ // `examples` and any other subkey are intentionally dropped: no binding
233+ // field references them, and they can be large.
234+ }
235+ }
236+
237+ return { text, headRanges, pathItems, schemas, smallComponentRanges };
238+};
239+
240+/** Strip exactly `indent` leading spaces from every line of `fragment`, lifting
241+ * an indent-N block back to column 0 so it parses as a standalone document. */
242+const dedent = (fragment: string, indent: number): string =>
243+ indent === 0 ? fragment : fragment.replace(new RegExp(`^ {1,${indent}}`, "gm"), "");
244+
245+const parseYaml = (text: string): unknown =>
246+ parseYamlDocument(text, { json: true, schema: JSON_SCHEMA });
247+
248+/**
249+ * Parse a single indent-N entry range (a path-item or a schema) in isolation.
250+ * Returns `[name, value]` where `name` is the entry's key and `value` its
251+ * parsed body, or null when the fragment does not parse to a single-key map.
252+ */
253+export const parseEntry = (
254+ text: string,
255+ range: ByteRange,
256+ indent: number,
257+): readonly [string, unknown] | null => {
258+ const parsed = parseYaml(dedent(text.slice(range.start, range.end), indent));
259+ if (parsed === null || typeof parsed !== "object" || Array.isArray(parsed)) return null;
260+ const entries = Object.entries(parsed as Record<string, unknown>);
261+ if (entries.length !== 1) return null;
262+ return entries[0]!;
263+};
264+
265+/** Parse the concatenation of the head ranges into the document head (openapi,
266+ * info, servers, tags). Small; safe to materialize whole. */
267+export const parseHead = (structure: SpecStructure): Record<string, unknown> => {
268+ const text = structure.headRanges.map((r) => structure.text.slice(r.start, r.end)).join("");
269+ const parsed = parseYaml(text);
270+ return parsed && typeof parsed === "object" && !Array.isArray(parsed)
271+ ? (parsed as Record<string, unknown>)
272+ : {};
273+};
274+
275+/** Parse the small component subkeys (parameters / requestBodies / responses)
276+ * into a `components`-shaped object for `$ref` resolution. */
277+export const parseSmallComponents = (structure: SpecStructure): Record<string, unknown> => {
278+ if (structure.smallComponentRanges.length === 0) return {};
279+ const body = structure.smallComponentRanges
280+ .map((r) => structure.text.slice(r.start, r.end))
281+ .join("");
282+ const parsed = parseYaml(`components:\n${body}`);
283+ const components =
284+ parsed && typeof parsed === "object" ? (parsed as Record<string, unknown>).components : null;
285+ return components && typeof components === "object"
286+ ? (components as Record<string, unknown>)
287+ : {};
288+};
289+
290+/**
291+ * Accept only the block-YAML profile the splitter can safely slice: no tabs, no
292+ * anchors/aliases, no merge keys, and a `paths:` block present. Block scalars
293+ * are allowed (tracked during the scan). Conservative by design: a false
294+ * negative just routes a spec through the whole-document parse.
295+ */
296+export const isStreamableSpec = (text: string): boolean => {
297+ if (text.indexOf("\t") !== -1) return false;
298+ // Anchors/aliases/merge keys: cheap substring rejects before structural work.
299+ if (/(^|\s)[&*][A-Za-z0-9_]/m.test(text)) return false;
300+ if (/<<\s*:/.test(text)) return false;
301+ return /^paths:/m.test(text);
302+};
303+
304+const SCHEMA_REF_PREFIX = "#/components/schemas/";
305+
306+/** Map each `components.schemas` entry name to its byte range, reading only the
307+ * key line (never parsing the schema body). The schema name is the raw YAML
308+ * key, which matches the trailing segment of a `#/components/schemas/<name>`
309+ * reference. */
310+export const indexSchemas = (structure: SpecStructure): ReadonlyMap<string, ByteRange> => {
311+ const index = new Map<string, ByteRange>();
312+ for (const range of structure.schemas) {
313+ const line = lineAt(structure.text, range.start, range.end);
314+ if (!line) continue;
315+ const name = keyNameAt(structure.text, range.start, line.lineEnd);
316+ if (name) index.set(name, range);
317+ }
318+ return index;
319+};
320+
321+/** Decode a `#/components/schemas/<segment>` name segment. Schema names are raw
322+ * YAML keys, so only JSON-pointer tilde escaping can appear (no `%`-encoding
323+ * in the wild specs we target); decode `~1`/`~0` and leave the rest verbatim. */
324+const decodeSchemaRefName = (segment: string): string =>
325+ segment.replace(/~1/g, "/").replace(/~0/g, "~");
326+
327+/** Collect the names of every `#/components/schemas/X` reference reachable in
328+ * `value`, without resolving them. */
329+const collectSchemaRefNames = (value: unknown, into: Set<string>): void => {
330+ if (typeof value === "string") {
331+ if (value.startsWith(SCHEMA_REF_PREFIX)) {
332+ const name = decodeSchemaRefName(value.slice(SCHEMA_REF_PREFIX.length));
333+ if (name.length > 0) into.add(name);
334+ }
335+ return;
336+ }
337+ if (Array.isArray(value)) {
338+ for (const item of value) collectSchemaRefNames(item, into);
339+ return;
340+ }
341+ if (value !== null && typeof value === "object") {
342+ for (const item of Object.values(value as Record<string, unknown>)) {
343+ collectSchemaRefNames(item, into);
344+ }
345+ }
346+};
347+
348+/**
349+ * Resolve the transitive closure of schemas referenced by `roots`, parsing each
350+ * referenced schema once from its byte range (BFS over `$ref`s). Schemas not
351+ * reachable from `roots` are never parsed, so peak memory tracks the kept
352+ * subset rather than the full `components.schemas` map.
353+ */
354+export const collectReferencedSchemas = (
355+ structure: SpecStructure,
356+ index: ReadonlyMap<string, ByteRange>,
357+ roots: readonly unknown[],
358+): Record<string, unknown> => {
359+ const result: Record<string, unknown> = {};
360+ const wanted = new Set<string>();
361+ for (const root of roots) collectSchemaRefNames(root, wanted);
362+
363+ const queue = [...wanted];
364+ for (let i = 0; i < queue.length; i += 1) {
365+ const name = queue[i]!;
366+ if (Object.prototype.hasOwnProperty.call(result, name)) continue;
367+ const range = index.get(name);
368+ if (!range) continue;
369+ const entry = parseEntry(structure.text, range, 4);
370+ if (!entry) continue;
371+ result[name] = entry[1];
372+ const next = new Set<string>();
373+ collectSchemaRefNames(entry[1], next);
374+ for (const ref of next) {
375+ if (!Object.prototype.hasOwnProperty.call(result, ref)) queue.push(ref);
376+ }
377+ }
378+ return result;
379+};
380+
381+/** A path-item filter for the streaming compile: given a parsed path-item,
382+ * return the (possibly trimmed) value to keep, or null to drop the path
383+ * entirely. Applied per path-item by `streamOperationBindingsFromStructure`. */
384+export type KeepPathItem = (
385+ path: string,
386+ pathItem: Record<string, unknown>,
387+) => Record<string, unknown> | null;