History

openapi: stream large specs so the full Microsoft Graph spec fits the Workers isolate (#1091)

* openapi: stream compile + serve from content-addressed bindings (no re-parse) The serve path (resolveOpenApiBackedTools, the 2nd OOM site) re-parsed the whole spec on every tools/list. For the full Microsoft Graph spec (37MB / 16.5k ops) that rebuilt a ~300MB document tree and OOM'd the 128MB Workers isolate even after the add-path parser swap. Add a memory-safe compile + serve path that never re-parses: - streamOperationBindings (extract.ts): two-pass streaming compile. Pass 1 plans tool paths from schema-free metadata (planToolPaths, extracted from compileToolDefinitions); pass 2 builds invocation bindings in bounded chunks, flushing each before the next so a huge spec's bindings are never all co-resident with the parsed tree. - Content-addressed defs blob (store.ts: putDefs/getDefs, keyed by specHash): the normalized #/$defs schemas, built once per spec at add time and shared across tenants. buildDefsJson serializes them one schema at a time so the normalized tree never co-resides with the parsed document. - Persist a resolved description per operation so the serve path rebuilds each tool def from the binding alone. - resolveOpenApiBackedTools fast path: getDefs(specHash) + listOperations + toolDefFromStoredOperation, falling back to a spec re-parse for legacy rows or a missing/corrupt blob. - Route the openapi, microsoft (full-graph), and google add/update paths through the streaming persist + putDefs. * e2e: guard full Graph catalog add + serve at real scale Drives the public API to add every Microsoft Graph workload (the full ~37MB / 16.5k-operation spec) and then list the served catalog. Pins both former OOM sites at real scale: the streaming add persists one binding per operation, and tools/list rebuilds >5000 tools from the persisted bindings plus the content-addressed defs blob, never re-parsing the spec. Proven green on cloud and selfhost. * openapi: keep the file-emit hint on the fast serve path The content-addressed serve path rebuilds a tool def from the persisted binding instead of re-parsing the spec. The re-parse path (kept as the fallback) appends the ToolFile emit contract to a file-returning tool's description at the projection step; the fast path bypassed that step and served the bare description, so a file-returning operation lost the contract whenever it was served from the binding (the common case once a spec is persisted). Apply the hint at the same projection step in the fast path, sourced from the binding's response fileHint (already inspected for the output schema), so a file tool carries the contract identically whether served fast or via the fallback. No persist-path or schema change: existing rows pick it up on the next list. Covered by the tool-descriptions e2e scenario. * openapi: stream large specs so the full Microsoft Graph spec fits the Workers isolate Adding the Microsoft Graph provider with the full spec (37MB, ~16,548 operations) 503'd on Cloudflare Workers. The cause is memory, not a bug: a whole-document YAML parse materializes a ~300MB JS tree that exceeds the 128MB-per-isolate limit, and the downstream compile pins that tree alive while it builds every operation, peaking far past the limit. Replace the whole-tree parse on the add/serve paths with a streaming, content-addressed compile that never builds the full tree: - split.ts: structuralSplit() extracts safe byte ranges from the constrained block-YAML profile (top-level keys at col 0, path-items at indent 2, schema entries at indent 4). It parses each path-item and each schema in isolation and discards it, so peak memory stays near a single item (~19MB on workerd) instead of ~300MB. Specs that do not present this profile are rejected with a precise diagnostic rather than silently falling back to the parse that OOMs. - extract.ts: streamOperationBindingsFromStructure() does a 2-pass, same-order compile. Pass 1 derives light schema-free metadata and plans tool paths (global collision resolution); pass 2 builds one binding per operation, flushes in chunks, and drops them. The pass-2 resolver is schema-free, so schema refs stay as lazy $ref strings. - backing.ts: compileAndPersistOpenApiSpecStreaming() streams bindings to storage and writes a content-addressed defs blob keyed by the source hash, shared across every tenant/selection on the same spec. invokeOpenApiBackedTool no longer re-parses the (now full) spec to enrich a present binding; it only re-parses when the binding is entirely absent (a legacy row). Wire the Microsoft Graph add/update path through the streaming primitive with a per-path-item keep filter for the scope selection, and delete the previous reduce-to-a-smaller-document model (and its unit test, superseded by the e2e scenario). Proven by e2e/scenarios/microsoft-graph-full.test.ts, which adds the full catalog and serves >5000 tools, green on both the cloudflare (workerd, the 128MB isolate that previously 503'd) and selfhost targets.

Rhys Sullivan committed 4 months ago

Commit 41137451 parent 108584410 files · +778 −641Browse files

Unified · Split

10 files
  1. e2e/vitest.config.ts +11 −4
  2. packages/plugins/microsoft/src/sdk/graph.test.ts +0 −326
  3. packages/plugins/microsoft/src/sdk/graph.ts +102 −278
  4. packages/plugins/microsoft/src/sdk/index.ts +1 −1
  5. packages/plugins/microsoft/src/sdk/plugin.test.ts +7 −6
  6. packages/plugins/microsoft/src/sdk/plugin.ts +17 −22
  7. packages/plugins/openapi/src/sdk/backing.ts +98 −2
  8. packages/plugins/openapi/src/sdk/extract.ts +140 −1
  9. packages/plugins/openapi/src/sdk/index.ts +15 −1
  10. packages/plugins/openapi/src/sdk/split.ts +387 −0
e2e/vitest.config.ts+11−4
@@ -39,11 +39,18 @@ export default defineConfig({ · 38 unmodified lines
39 fileParallelism: false,39 fileParallelism: false,
40 }),40 }),
41 // The Cloudflare self-host worker (workerd via wrangler dev, dev-auth).41 // The Cloudflare self-host worker (workerd via wrangler dev, dev-auth).
42 // Scoped to the browser-approval scenario for now — the only cross-target42 // Scoped to the cross-target scenarios wired for this host; the rest of
43 // scenario wired for this host; the rest of scenarios/** is not yet43 // scenarios/** is not yet validated against the worker. The full-graph
44 // validated against the worker. Shares self-host's single-admin model.44 // scenario is included on purpose: workerd's 128MB isolate is the exact
45 // limit the streaming compile + content-addressed serve path defends, so
46 // it is the one host where that regression must be proven. Shares
47 // self-host's single-admin model.
45 project("cloudflare", {48 project("cloudflare", {
46 include: ["scenarios/browser-approval.test.ts", "cloudflare/**/*.test.ts"],49 include: [
50 "scenarios/browser-approval.test.ts",
51 "scenarios/microsoft-graph-full.test.ts",
52 "cloudflare/**/*.test.ts",
53 ],
47 fileParallelism: false,54 fileParallelism: false,
48 }),55 }),
49 // The Electron desktop app. Only desktop/** scenarios — the desktop56 // The Electron desktop app. Only desktop/** scenarios — the desktop
packages/plugins/microsoft/src/sdk/graph.test.ts+0−326
@@ -1,326 +0,0 @@
1import { describe, expect, it } from "@effect/vitest";
2import { Effect, Schema } from "effect";
3
4import { MICROSOFT_AUTH_TEMPLATE_SLUG, MICROSOFT_GRAPH_CLIENT_CREDENTIALS_SCOPES } from "./presets";
5import {
6 buildFilteredMicrosoftGraphOpenApiSpec,
7 filterMicrosoftGraphOpenApiSpec,
8 parseMicrosoftGraphDelegatedScopes,
9} from "./graph";
10
11// The filtered preset spec is emitted as JSON, so re-read it with Effect Schema
12// rather than a YAML parser.
13const decodeJson = Schema.decodeUnknownSync(Schema.fromJsonString(Schema.Unknown));
14
15const graphFixture = `
16openapi: 3.0.4
17info:
18 title: Microsoft Graph Fixture
19 version: v1.0
20servers:
21 - url: https://graph.microsoft.com/v1.0
22paths:
23 /me:
24 get:
25 operationId: me.GetUser
26 tags:
27 - me.user
28 responses:
29 "200":
30 description: OK
31 /me/messages:
32 get:
33 operationId: me.messages.ListMessages
34 tags:
35 - me.message
36 security:
37 - azureAdDelegated:
38 - Mail.ReadWrite
39 responses:
40 "200":
41 description: OK
42 content:
43 application/json:
44 schema:
45 $ref: "#/components/schemas/messageCollection"
46 /me/onenote/pages:
47 get:
48 operationId: me.onenote.pages.ListPages
49 tags:
50 - me.onenote
51 security:
52 - azureAdDelegated:
53 - Notes.ReadWrite
54 responses:
55 "200":
56 description: OK
57 /teams/{team-id}/channels/{channel-id}/messages:
58 post:
59 operationId: teams.channels.messages.CreateMessage
60 tags:
61 - teams.channel
62 x-ms-permissions:
63 delegated:
64 - ChannelMessage.Send
65 responses:
66 "200":
67 description: OK
68 /sites:
69 get:
70 operationId: sites.ListSites
71 tags:
72 - sites.site
73 responses:
74 "200":
75 description: OK
76components:
77 schemas:
78 user:
79 type: object
80 messageCollection:
81 type: object
82 properties:
83 value:
84 type: array
85 items:
86 $ref: "#/components/schemas/message"
87 message:
88 type: object
89 properties:
90 id:
91 type: string
92 unusedDirectoryObject:
93 type: object
94`;
95
96describe("Microsoft Graph OpenAPI filtering", () => {
97 it("parses delegated scopes from the generated Microsoft permissions reference", () => {
98 expect(
99 parseMicrosoftGraphDelegatedScopes(`
100### User.Read
101
102| Category | Application | Delegated |
103|--|--|--|
104| Identifier | - | e1fe6dd8-ba31-4d61-89e7-88639da4683d |
105
106---
107
108### AppCatalog.Read.All
109
110| Category | Application | Delegated |
111|--|--|--|
112| Identifier | e12dae10-5a57-4817-b79d-dfbec5348930 | - |
113
114---
115
116### Calendars.ReadWrite
117
118| Category | Application | Delegated |
119|--|--|--|
120| Identifier | ef54d2bf-783f-4e0f-bca1-3210c0444d99 | 1ec239c2-d7c9-4623-a91a-a9775856bb36 |
121`),
122 ).toEqual(["User.Read", "Calendars.ReadWrite"]);
123 });
124
125 it.effect("keeps selected paths and injects delegated OAuth", () =>
126 Effect.gen(function* () {
127 const filtered = yield* filterMicrosoftGraphOpenApiSpec(graphFixture, {
128 scopes: ["offline_access", "User.Read", "Mail.ReadWrite"],
129 exactPaths: ["/me"],
130 pathPrefixes: ["/me/messages"],
131 tagPrefixes: [],
132 });
133 const doc = decodeJson(filtered) as {
134 readonly paths: Record<string, unknown>;
135 readonly components: {
136 readonly securitySchemes: Record<string, unknown>;
137 };
138 readonly security: readonly Record<string, readonly string[]>[];
139 };
140
141 expect(Object.keys(doc.paths).sort()).toEqual(["/me", "/me/messages"]);
142 expect(doc.components.securitySchemes[MICROSOFT_AUTH_TEMPLATE_SLUG]).toBeDefined();
143 expect(doc.security[0]?.[MICROSOFT_AUTH_TEMPLATE_SLUG]).toEqual([
144 "offline_access",
145 "User.Read",
146 "Mail.ReadWrite",
147 ]);
148 }),
149 );
150
151 it.effect("prunes unreferenced components from filtered specs", () =>
152 Effect.gen(function* () {
153 const filtered = yield* filterMicrosoftGraphOpenApiSpec(graphFixture, {
154 scopes: ["offline_access", "Mail.ReadWrite"],
155 exactPaths: [],
156 pathPrefixes: ["/me/messages"],
157 tagPrefixes: [],
158 });
159 const doc = decodeJson(filtered) as {
160 readonly components: {
161 readonly schemas?: Record<string, unknown>;
162 readonly securitySchemes: Record<string, unknown>;
163 };
164 };
165
166 expect(Object.keys(doc.components.schemas ?? {}).sort()).toEqual([
167 "message",
168 "messageCollection",
169 ]);
170 expect(doc.components.schemas?.unusedDirectoryObject).toBeUndefined();
171 expect(doc.components.securitySchemes[MICROSOFT_AUTH_TEMPLATE_SLUG]).toBeDefined();
172 }),
173 );
174
175 it.effect("keeps operations matched by selected Graph scopes", () =>
176 Effect.gen(function* () {
177 const filtered = yield* filterMicrosoftGraphOpenApiSpec(graphFixture, {
178 scopes: ["offline_access", "Notes.ReadWrite", "ChannelMessage.Send"],
179 exactPaths: [],
180 pathPrefixes: [],
181 tagPrefixes: [],
182 });
183 const doc = decodeJson(filtered) as {
184 readonly paths: Record<string, unknown>;
185 };
186
187 expect(Object.keys(doc.paths).sort()).toEqual([
188 "/me/onenote/pages",
189 "/teams/{team-id}/channels/{channel-id}/messages",
190 ]);
191 }),
192 );
193
194 it.effect("keeps operations matched by selected Graph tags", () =>
195 Effect.gen(function* () {
196 const filtered = yield* filterMicrosoftGraphOpenApiSpec(graphFixture, {
197 scopes: ["offline_access"],
198 exactPaths: [],
199 pathPrefixes: [],
200 tagPrefixes: ["teams."],
201 });
202 const doc = decodeJson(filtered) as {
203 readonly paths: Record<string, unknown>;
204 };
205
206 expect(Object.keys(doc.paths)).toEqual(["/teams/{team-id}/channels/{channel-id}/messages"]);
207 }),
208 );
209
210 it.effect("keeps categorized Graph selections and derives delegated scopes from the spec", () =>
211 Effect.gen(function* () {
212 const filtered = yield* buildFilteredMicrosoftGraphOpenApiSpec(graphFixture, {
213 scopes: ["offline_access"],
214 exactPaths: [],
215 pathPrefixes: ["/me", "/sites", "/teams"],
216 tagPrefixes: [],
217 fullGraphScopes: ["User.Read", "Mail.ReadWrite", "Notes.ReadWrite"],
218 });
219 const doc = decodeJson(filtered.specText) as {
220 readonly paths: Record<string, unknown>;
221 readonly security: readonly Record<string, readonly string[]>[];
222 };
223
224 expect(Object.keys(doc.paths).sort()).toEqual([
225 "/me",
226 "/me/messages",
227 "/me/onenote/pages",
228 "/sites",
229 "/teams/{team-id}/channels/{channel-id}/messages",
230 ]);
231 expect(filtered.scopes).toEqual([
232 "offline_access",
233 "User.Read",
234 "Mail.ReadWrite",
235 "Notes.ReadWrite",
236 "ChannelMessage.Send",
237 ]);
238 expect(doc.security[0]?.[MICROSOFT_AUTH_TEMPLATE_SLUG]).toEqual(filtered.scopes);
239 }),
240 );
241
242 it.effect("keeps emulator versioned paths and preserves emulator OAuth endpoints", () =>
243 Effect.gen(function* () {
244 const filtered = yield* filterMicrosoftGraphOpenApiSpec(
245 `
246openapi: 3.0.3
247info:
248 title: Microsoft Graph Emulator
249 version: 1.0.0
250servers:
251 - url: https://microsoft.emulators.dev
252paths:
253 /v1.0/me:
254 get:
255 operationId: graphUser_GetMyProfile
256 responses:
257 "200":
258 description: OK
259 /v1.0/users:
260 get:
261 operationId: graphUser_List
262 responses:
263 "200":
264 description: OK
265components:
266 securitySchemes:
267 azureAdDelegated:
268 type: oauth2
269 flows:
270 authorizationCode:
271 authorizationUrl: https://microsoft.emulators.dev/oauth2/v2.0/authorize
272 tokenUrl: https://microsoft.emulators.dev/oauth2/v2.0/token
273 scopes:
274 User.Read: User.Read
275 clientCredentials:
276 tokenUrl: https://microsoft.emulators.dev/oauth2/v2.0/token
277 scopes:
278 https://graph.microsoft.com/.default: https://graph.microsoft.com/.default
279`,
280 {
281 scopes: ["offline_access", "User.Read", "User.Read.All"],
282 exactPaths: ["/me"],
283 pathPrefixes: ["/users"],
284 tagPrefixes: [],
285 },
286 );
287 const doc = decodeJson(filtered) as {
288 readonly servers: readonly { readonly url: string }[];
289 readonly paths: Record<string, unknown>;
290 readonly components: {
291 readonly securitySchemes: Record<
292 string,
293 {
294 readonly flows: {
295 readonly authorizationCode: {
296 readonly authorizationUrl: string;
297 readonly tokenUrl: string;
298 };
299 readonly clientCredentials: {
300 readonly tokenUrl: string;
301 readonly scopes: Record<string, string>;
302 };
303 };
304 }
305 >;
306 };
307 };
308
309 expect(doc.servers[0]?.url).toBe("https://microsoft.emulators.dev");
310 expect(Object.keys(doc.paths).sort()).toEqual(["/v1.0/me", "/v1.0/users"]);
311 const flows = doc.components.securitySchemes[MICROSOFT_AUTH_TEMPLATE_SLUG]?.flows;
312 expect(flows?.authorizationCode.authorizationUrl).toBe(
313 "https://microsoft.emulators.dev/oauth2/v2.0/authorize",
314 );
315 expect(flows?.authorizationCode.tokenUrl).toBe(
316 "https://microsoft.emulators.dev/oauth2/v2.0/token",
317 );
318 expect(flows?.clientCredentials.tokenUrl).toBe(
319 "https://microsoft.emulators.dev/oauth2/v2.0/token",
320 );
321 expect(Object.keys(flows?.clientCredentials.scopes ?? {})).toEqual([
322 ...MICROSOFT_GRAPH_CLIENT_CREDENTIALS_SCOPES,
323 ]);
324 }),
325 );
326});
packages/plugins/microsoft/src/sdk/graph.ts+102−278
@@ -6,17 +6,20 @@ import { AuthTemplateSlug } from "@executor-js/sdk/core"; · 5 unmodified lines
6import {6import {
7 AuthenticationSchema,7 AuthenticationSchema,
8 OpenApiParseError,8 OpenApiParseError,
9 parse as parseOpenApiDocument,9 parseEntry,
10 parseHead,
11 parseSmallComponents,
12 structuralSplit,
10 type Authentication,13 type Authentication,
14 type KeepPathItem,
11 type OpenApiIntegrationConfig,15 type OpenApiIntegrationConfig,
12 type ParsedDocument,16 type SpecStructure,
13} from "@executor-js/plugin-openapi";17} from "@executor-js/plugin-openapi";
14 18
15import {19import {
16 MICROSOFT_AUTHORIZATION_URL,20 MICROSOFT_AUTHORIZATION_URL,
17 MICROSOFT_AUTH_TEMPLATE_SLUG,21 MICROSOFT_AUTH_TEMPLATE_SLUG,
18 MICROSOFT_CLIENT_CREDENTIALS_AUTH_TEMPLATE_SLUG,22 MICROSOFT_CLIENT_CREDENTIALS_AUTH_TEMPLATE_SLUG,
19 MICROSOFT_GRAPH_BASE_URL,
20 MICROSOFT_GRAPH_BASE_SCOPES,23 MICROSOFT_GRAPH_BASE_SCOPES,
21 MICROSOFT_GRAPH_CLIENT_CREDENTIALS_SCOPES,24 MICROSOFT_GRAPH_CLIENT_CREDENTIALS_SCOPES,
22 MICROSOFT_GRAPH_DELEGATED_DEFAULT_SCOPES,25 MICROSOFT_GRAPH_DELEGATED_DEFAULT_SCOPES,
@@ -43,7 +46,6 @@ export interface MicrosoftGraphSelectionInput { · 20 unmodified lines
43 46
44export interface MicrosoftGraphSpecBuild {47export interface MicrosoftGraphSpecBuild {
45 readonly specText: string;48 readonly specText: string;
46 readonly parsedDocument?: ParsedDocument;
47 readonly specUrl: string;49 readonly specUrl: string;
48 readonly baseUrl?: string;50 readonly baseUrl?: string;
49 readonly authorizationUrl: string;51 readonly authorizationUrl: string;
@@ -93,8 +95,6 @@ const MicrosoftGraphIntegrationConfigSchema = Schema.Struct({ · 43 unmodified lines
93 95
94const decodeMicrosoftConfig = Schema.decodeUnknownOption(MicrosoftGraphIntegrationConfigSchema);96const decodeMicrosoftConfig = Schema.decodeUnknownOption(MicrosoftGraphIntegrationConfigSchema);
95 97
96type MicrosoftGraphOpenApiDocument = ParsedDocument & Record<string, unknown>;
97
98export const decodeMicrosoftGraphIntegrationConfig = (98export const decodeMicrosoftGraphIntegrationConfig = (
99 value: unknown,99 value: unknown,
100): MicrosoftGraphIntegrationConfig | null =>100): MicrosoftGraphIntegrationConfig | null =>
@@ -174,10 +174,6 @@ const microsoftOAuthTemplate = ( · 73 unmodified lines
174const isRecord = (value: unknown): value is Record<string, unknown> =>174const isRecord = (value: unknown): value is Record<string, unknown> =>
175 value !== null && typeof value === "object" && !Array.isArray(value);175 value !== null && typeof value === "object" && !Array.isArray(value);
176 176
177// Hoisted so the compiled encoder is built once, not per call. Emits the small
178// filtered preset spec as JSON, which the OpenAPI SDK's `parse` reads back.
179const encodeOpenApiDocumentToJson = Schema.encodeSync(Schema.fromJsonString(Schema.Unknown));
180
181const HTTP_METHODS = new Set(["delete", "get", "patch", "post", "put"]);177const HTTP_METHODS = new Set(["delete", "get", "patch", "post", "put"]);
182const BASE_OAUTH_SCOPES = new Set(["offline_access", "openid", "profile", "email"]);178const BASE_OAUTH_SCOPES = new Set(["offline_access", "openid", "profile", "email"]);
183 179
@@ -187,17 +183,6 @@ const firstString = (values: readonly unknown[]): string | undefined => · 3 unmodified lines
187const recordValues = (value: unknown): readonly unknown[] =>183const recordValues = (value: unknown): readonly unknown[] =>
188 isRecord(value) ? Object.values(value) : [];184 isRecord(value) ? Object.values(value) : [];
189 185
190const firstServerUrl = (parsed: Record<string, unknown>): string | undefined => {
191 const servers = parsed.servers;
192 if (!Array.isArray(servers)) return undefined;
193 for (const server of servers) {
194 if (!isRecord(server)) continue;
195 const url = server.url;
196 if (typeof url === "string" && url.trim().length > 0) return url.trim();
197 }
198 return undefined;
199};
200
201const firstOAuthFlows = (parsed: Record<string, unknown>): readonly Record<string, unknown>[] => {186const firstOAuthFlows = (parsed: Record<string, unknown>): readonly Record<string, unknown>[] => {
202 const components = isRecord(parsed.components) ? parsed.components : {};187 const components = isRecord(parsed.components) ? parsed.components : {};
203 const securitySchemes = isRecord(components.securitySchemes) ? components.securitySchemes : {};188 const securitySchemes = isRecord(components.securitySchemes) ? components.securitySchemes : {};
@@ -336,25 +321,6 @@ const operationMatchesScope = ( · 132 unmodified lines
336 (scope) => selectedScopes.has(scope) && !BASE_OAUTH_SCOPES.has(scope),321 (scope) => selectedScopes.has(scope) && !BASE_OAUTH_SCOPES.has(scope),
337 );322 );
338 323
339const selectedOAuthScopesForPaths = (
340 paths: Record<string, unknown>,
341 requestedScopes: readonly string[],
342 fullGraphScopes: readonly string[] = [],
343): readonly string[] =>
344 uniqueStrings([
345 ...MICROSOFT_GRAPH_BASE_SCOPES,
346 ...fullGraphScopes,
347 ...requestedScopes.filter((scope) => !BASE_OAUTH_SCOPES.has(scope)),
348 ...Object.values(paths).flatMap((pathItem) => {
349 if (!isRecord(pathItem)) return [];
350 return Object.entries(pathItem).flatMap(([method, operation]) =>
351 HTTP_METHODS.has(method.toLowerCase()) && isRecord(operation)
352 ? permissionScopes(operation, { delegatedOnly: true })
353 : [],
354 );
355 }),
356 ]);
357
358const filterPathItem = (324const filterPathItem = (
359 path: string,325 path: string,
360 pathItem: Record<string, unknown>,326 pathItem: Record<string, unknown>,
@@ -390,89 +356,6 @@ const filterPathItem = ( · 29 unmodified lines
390 return kept;356 return kept;
391};357};
392 358
393const selectMicrosoftGraphPaths = (
394 paths: Record<string, unknown>,
395 options: {
396 readonly scopes: readonly string[];
397 readonly exactPaths: readonly string[];
398 readonly pathPrefixes: readonly string[];
399 readonly tagPrefixes: readonly string[];
400 },
401): Record<string, unknown> => {
402 const exactPaths = new Set(options.exactPaths);
403 const selectedScopes = new Set(options.scopes);
404 const entries = Object.entries(paths).flatMap(([path, pathItem]) => {
405 if (!isRecord(pathItem)) return [];
406 const filtered = filterPathItem(path, pathItem, {
407 exactPaths,
408 pathPrefixes: options.pathPrefixes,
409 tagPrefixes: options.tagPrefixes,
410 selectedScopes,
411 });
412 return filtered ? ([[path, filtered]] as const) : [];
413 });
414 return Object.fromEntries(entries);
415};
416
417const decodeJsonPointerSegment = (segment: string): string =>
418 decodeURIComponent(segment).replace(/~1/g, "/").replace(/~0/g, "~");
419
420const componentRefFromString = (
421 value: string,
422): { readonly section: string; readonly name: string } | null => {
423 if (!value.startsWith("#/components/")) return null;
424 const [, section, ...nameSegments] = value.slice(2).split("/");
425 if (!section || nameSegments.length === 0) return null;
426 return {
427 section: decodeJsonPointerSegment(section),
428 name: nameSegments.map(decodeJsonPointerSegment).join("/"),
429 };
430};
431
432const collectComponentRefs = (
433 value: unknown,
434): readonly { readonly section: string; readonly name: string }[] => {
435 if (typeof value === "string") {
436 const ref = componentRefFromString(value);
437 return ref ? [ref] : [];
438 }
439 if (Array.isArray(value)) return value.flatMap(collectComponentRefs);
440 if (!isRecord(value)) return [];
441 return Object.values(value).flatMap(collectComponentRefs);
442};
443
444const prunedComponentsForPaths = (
445 components: Record<string, unknown>,
446 paths: Record<string, unknown>,
447): Record<string, unknown> => {
448 const pruned: Record<string, Record<string, unknown>> = {};
449 const queue = [...collectComponentRefs(paths)];
450 const seen = new Set<string>();
451
452 for (let index = 0; index < queue.length; index += 1) {
453 const ref = queue[index];
454 const key = `${ref.section}/${ref.name}`;
455 if (seen.has(key)) continue;
456 seen.add(key);
457
458 const section = components[ref.section];
459 if (!isRecord(section)) continue;
460 const component = section[ref.name];
461 if (component === undefined) continue;
462
463 pruned[ref.section] ??= {};
464 pruned[ref.section][ref.name] = component;
465 queue.push(...collectComponentRefs(component));
466 }
467
468 return pruned;
469};
470
471interface MicrosoftGraphFilterResult {
472 readonly specText: string;
473 readonly scopes: readonly string[];
474}
475
476export const fetchMicrosoftGraphOpenApiSpec = Effect.fn("Microsoft.fetchGraphOpenApiSpec")(359export const fetchMicrosoftGraphOpenApiSpec = Effect.fn("Microsoft.fetchGraphOpenApiSpec")(
477 function* (specUrl: string) {360 function* (specUrl: string) {
478 const client = yield* HttpClient.HttpClient;361 const client = yield* HttpClient.HttpClient;
@@ -539,128 +422,70 @@ export const fetchMicrosoftGraphPermissionsReference = Effect.fn( · 60 unmodified lines
539 );422 );
540});423});
541 424
542const parseMicrosoftGraphOpenApiDocument = (425/**
543 specText: string,426 * Build the per-path-item filter that the streaming compile applies to each
544): Effect.Effect<MicrosoftGraphOpenApiDocument, OpenApiParseError> =>427 * path-item as it parses the 37MB source. Returns `undefined` for a full-graph
545 // Delegate to the OpenAPI SDK's lean js-yaml parser. The bespoke eemeli/yaml428 * selection (keep everything). The selection predicate is identical to the old
546 // parser this replaced built a full CST that peaks ~720MB heap on the 37MB429 * two-pass filter: the selected scopes are derived from the PRESET scopes
547 // Graph spec and OOMs the 128MB Cloudflare Workers isolate (measured: HTTP430 * (`microsoftGraphScopesForPresetIds`), not the expanded OAuth scopes, so the
548 // 503, outcome=exceededMemory). The SDK parser peaks ~315MB and runs in-band431 * kept operation set matches regardless of caller.
549 // on Workers. `parse` already validates an OpenAPI 3.x object and does not432 */
550 // inline `$ref`s, so peak memory stays bounded for big specs.433export const microsoftGraphKeepPathItem = (selection: {
551 parseOpenApiDocument(specText).pipe(434 readonly coversFullGraph: boolean;
552 Effect.mapError(435 readonly presetIds: readonly string[];
553 () =>436 readonly customScopes: readonly string[];
554 new OpenApiParseError({437 readonly exactPaths: readonly string[];
555 message: "Failed to parse Microsoft Graph OpenAPI document",438 readonly pathPrefixes: readonly string[];
556 }),439 readonly tagPrefixes: readonly string[];
557 ),440}): KeepPathItem | undefined => {
558 Effect.map((doc) => doc as MicrosoftGraphOpenApiDocument),441 if (selection.coversFullGraph) return undefined;
442 const exactPaths = new Set(selection.exactPaths);
443 const selectedScopes = new Set(
444 microsoftGraphScopesForPresetIds(selection.presetIds, selection.customScopes),
559 );445 );
560 446 return (path, pathItem) =>
561export const buildFilteredMicrosoftGraphOpenApiSpecFromDocument = (447 filterPathItem(path, pathItem, {
562 parsed: MicrosoftGraphOpenApiDocument,448 exactPaths,
563 options: {449 pathPrefixes: selection.pathPrefixes,
564 readonly scopes: readonly string[];450 tagPrefixes: selection.tagPrefixes,
565 readonly exactPaths: readonly string[];451 selectedScopes,
566 readonly pathPrefixes: readonly string[];
567 readonly tagPrefixes: readonly string[];
568 readonly baseUrl?: string;
569 readonly authorizationUrl?: string;
570 readonly tokenUrl?: string;
571 readonly clientCredentialsTokenUrl?: string;
572 readonly fullGraphScopes?: readonly string[];
573 },
574): Effect.Effect<MicrosoftGraphFilterResult, OpenApiParseError> =>
575 Effect.gen(function* () {
576 const paths = parsed.paths;
577 if (!isRecord(paths)) {
578 return yield* new OpenApiParseError({
579 message: "Microsoft Graph OpenAPI document is missing paths",
580 });
581 }
582
583 const filteredPaths = selectMicrosoftGraphPaths(paths, options);
584 if (Object.keys(filteredPaths).length === 0) {
585 return yield* new OpenApiParseError({
586 message: "Microsoft Graph scope selection did not match any OpenAPI paths",
587 });
588 }
589
590 const scopes = selectedOAuthScopesForPaths(
591 filteredPaths,
592 options.scopes,
593 options.fullGraphScopes ?? [],
594 );
595 const serverUrl = options.baseUrl ?? firstServerUrl(parsed) ?? MICROSOFT_GRAPH_BASE_URL;
596 const endpoints = resolveOAuthEndpoints(parsed, options);
597 const components = isRecord(parsed.components) ? parsed.components : {};
598 const securitySchemes = isRecord(components.securitySchemes) ? components.securitySchemes : {};
599 const filteredComponents = prunedComponentsForPaths(components, filteredPaths);
600 const next = {
601 ...parsed,
602 info: {
603 ...(isRecord(parsed.info) ? parsed.info : {}),
604 title: "Microsoft Graph",
605 description: "Selected Microsoft Graph workloads from the v1.0 OpenAPI document.",
606 },
607 servers: [{ url: serverUrl }],
608 paths: filteredPaths,
609 components: {
610 ...filteredComponents,
611 securitySchemes: {
612 ...securitySchemes,
613 [MICROSOFT_AUTH_TEMPLATE_SLUG]: {
614 type: "oauth2",
615 flows: {
616 authorizationCode: {
617 authorizationUrl: endpoints.authorizationUrl,
618 tokenUrl: endpoints.tokenUrl,
619 scopes: Object.fromEntries(scopes.map((scope) => [scope, scope])),
620 },
621 clientCredentials: {
622 tokenUrl: endpoints.clientCredentialsTokenUrl,
623 scopes: Object.fromEntries(
624 MICROSOFT_GRAPH_CLIENT_CREDENTIALS_SCOPES.map((scope) => [scope, scope]),
625 ),
626 },
627 },
628 },
629 },
630 },
631 security: [{ [MICROSOFT_AUTH_TEMPLATE_SLUG]: [...scopes] }],
632 };
633
634 // Serialize as JSON, not YAML. The OpenAPI SDK's `parse` reads JSON (a YAML
635 // subset) on the way back in, so the filtered preset spec round-trips without
636 // pulling in a second YAML library. This is the small filtered spec, never
637 // the 37MB source document.
638 const filteredSpecText = yield* Effect.try({
639 try: () => encodeOpenApiDocumentToJson(next),
640 catch: () =>
641 new OpenApiParseError({
642 message: "Failed to serialize Microsoft Graph OpenAPI document",
643 }),
644 });452 });
645 return { specText: filteredSpecText, scopes };453};
646 });
647
648export const buildFilteredMicrosoftGraphOpenApiSpec = (
649 specText: string,
650 options: Parameters<typeof buildFilteredMicrosoftGraphOpenApiSpecFromDocument>[1],
651): Effect.Effect<MicrosoftGraphFilterResult, OpenApiParseError> =>
652 Effect.gen(function* () {
653 const parsed = yield* parseMicrosoftGraphOpenApiDocument(specText);
654 return yield* buildFilteredMicrosoftGraphOpenApiSpecFromDocument(parsed, options);
655 });
656 454
657export const filterMicrosoftGraphOpenApiSpec = (455/**
658 specText: string,456 * Compute the OAuth scopes for the selection by streaming the source path-items
659 options: Parameters<typeof buildFilteredMicrosoftGraphOpenApiSpec>[1],457 * once (never materializing the whole tree). Mirrors the old
660): Effect.Effect<string, OpenApiParseError> =>458 * `selectedOAuthScopesForPaths`: base scopes + full-graph scopes + requested
661 buildFilteredMicrosoftGraphOpenApiSpec(specText, options).pipe(459 * scopes + the delegated permission scopes of every kept operation. `keepPathItem`
662 Effect.map((result) => result.specText),460 * (when present) restricts the walk to the filtered operation set, exactly as the
663 );461 * old code computed scopes over the already-filtered paths.
462 */
463const streamSelectedScopes = (
464 structure: SpecStructure,
465 requestedScopes: readonly string[],
466 fullGraphScopes: readonly string[],
467 keepPathItem?: KeepPathItem,
468): readonly string[] => {
469 const collected = [
470 ...MICROSOFT_GRAPH_BASE_SCOPES,
471 ...fullGraphScopes,
472 ...requestedScopes.filter((scope) => !BASE_OAUTH_SCOPES.has(scope)),
473 ];
474 for (const range of structure.pathItems) {
475 const entry = parseEntry(structure.text, range, 2);
476 if (!entry) continue;
477 const [path, rawItem] = entry;
478 if (!isRecord(rawItem)) continue;
479 const pathItem = keepPathItem ? keepPathItem(path, rawItem) : rawItem;
480 if (!pathItem) continue;
481 for (const [method, operation] of Object.entries(pathItem)) {
482 if (HTTP_METHODS.has(method.toLowerCase()) && isRecord(operation)) {
483 collected.push(...permissionScopes(operation, { delegatedOnly: true }));
484 }
485 }
486 }
487 return uniqueStrings(collected);
488};
664 489
665export const buildMicrosoftGraphOpenApiSpec = (490export const buildMicrosoftGraphOpenApiSpec = (
666 input: MicrosoftGraphSelectionInput,491 input: MicrosoftGraphSelectionInput,
@@ -671,6 +496,24 @@ export const buildMicrosoftGraphOpenApiSpec = ( · 4 unmodified lines
671 const sourceText = yield* fetchMicrosoftGraphOpenApiSpec(selection.specUrl).pipe(496 const sourceText = yield* fetchMicrosoftGraphOpenApiSpec(selection.specUrl).pipe(
672 Effect.provide(httpClientLayer),497 Effect.provide(httpClientLayer),
673 );498 );
499
500 // Structural split is the only entry point: parsing the whole 37MB tree
501 // OOMs the 128MB Workers isolate (measured: HTTP 503). No fallback. A spec
502 // outside the streamable block-YAML profile is a hard error on this path;
503 // arbitrary user specs still go through the generic openapi plugin.
504 const structure = structuralSplit(sourceText);
505 if (!structure) {
506 return yield* new OpenApiParseError({
507 message:
508 "Microsoft Graph OpenAPI document is not in the streamable block-YAML profile; cannot compile it in-band on Workers.",
509 });
510 }
511
512 // Head + small components (servers + securitySchemes) parse cheaply and
513 // carry everything `resolveOAuthEndpoints` needs.
514 const headDoc = { ...parseHead(structure), components: parseSmallComponents(structure) };
515 const endpoints = resolveOAuthEndpoints(headDoc, selection);
516
674 const permissionsReference =517 const permissionsReference =
675 selection.coversFullGraph === true518 selection.coversFullGraph === true
676 ? yield* fetchMicrosoftGraphPermissionsReference().pipe(Effect.provide(httpClientLayer))519 ? yield* fetchMicrosoftGraphPermissionsReference().pipe(Effect.provide(httpClientLayer))
@@ -678,46 +521,27 @@ export const buildMicrosoftGraphOpenApiSpec = ( · 1 unmodified line
678 const fullGraphScopes = permissionsReference521 const fullGraphScopes = permissionsReference
679 ? parseMicrosoftGraphDelegatedScopes(permissionsReference)522 ? parseMicrosoftGraphDelegatedScopes(permissionsReference)
680 : [];523 : [];
681 const parsed = yield* parseMicrosoftGraphOpenApiDocument(sourceText);524
682 const endpoints = resolveOAuthEndpoints(parsed, selection);525 const keepPathItem = microsoftGraphKeepPathItem(selection);
683 const graphPaths = parsed.paths;526 const scopes =
684 if (!isRecord(graphPaths)) {527 selection.coversFullGraph === true && selection.customScopes.length === 0
685 return yield* new OpenApiParseError({528 ? [...MICROSOFT_GRAPH_DELEGATED_DEFAULT_SCOPES]
686 message: "Microsoft Graph OpenAPI document is missing paths",529 : streamSelectedScopes(
687 });530 structure,
688 }531 selection.coversFullGraph === true
689 if (selection.coversFullGraph === true) {532 ? uniqueStrings([...MICROSOFT_GRAPH_BASE_SCOPES, ...selection.customScopes])
690 const scopes =533 : selection.scopes,
691 selection.customScopes.length === 0534 fullGraphScopes,
692 ? MICROSOFT_GRAPH_DELEGATED_DEFAULT_SCOPES535 keepPathItem,
693 : selectedOAuthScopesForPaths(536 );
694 graphPaths,537
695 uniqueStrings([...MICROSOFT_GRAPH_BASE_SCOPES, ...selection.customScopes]),
696 fullGraphScopes,
697 );
698 return {
699 ...selection,
700 specText: sourceText,
701 parsedDocument: parsed,
702 scopes,
703 authorizationUrl: endpoints.authorizationUrl,
704 tokenUrl: endpoints.tokenUrl,
705 clientCredentialsTokenUrl: endpoints.clientCredentialsTokenUrl,
706 authenticationTemplate: microsoftOAuthTemplate(scopes, endpoints),
707 };
708 }
709 const filtered = yield* buildFilteredMicrosoftGraphOpenApiSpecFromDocument(parsed, {
710 ...selection,
711 scopes: selection.scopes,
712 fullGraphScopes,
713 });
714 return {538 return {
715 ...selection,539 ...selection,
716 specText: filtered.specText,540 specText: sourceText,
717 scopes: filtered.scopes,541 scopes,
718 authorizationUrl: endpoints.authorizationUrl,542 authorizationUrl: endpoints.authorizationUrl,
719 tokenUrl: endpoints.tokenUrl,543 tokenUrl: endpoints.tokenUrl,
720 clientCredentialsTokenUrl: endpoints.clientCredentialsTokenUrl,544 clientCredentialsTokenUrl: endpoints.clientCredentialsTokenUrl,
721 authenticationTemplate: microsoftOAuthTemplate(filtered.scopes, endpoints),545 authenticationTemplate: microsoftOAuthTemplate(scopes, endpoints),
722 };546 };
723 });547 });
packages/plugins/microsoft/src/sdk/index.ts+1−1
@@ -30,7 +30,7 @@ export { · 29 unmodified lines
30 decodeMicrosoftGraphIntegrationConfig,30 decodeMicrosoftGraphIntegrationConfig,
31 fetchMicrosoftGraphOpenApiSpec,31 fetchMicrosoftGraphOpenApiSpec,
32 fetchMicrosoftGraphPermissionsReference,32 fetchMicrosoftGraphPermissionsReference,
33 filterMicrosoftGraphOpenApiSpec,33 microsoftGraphKeepPathItem,
34 parseMicrosoftGraphDelegatedScopes,34 parseMicrosoftGraphDelegatedScopes,
35 type MicrosoftGraphIntegrationConfig,35 type MicrosoftGraphIntegrationConfig,
36 type MicrosoftGraphSelectionInput,36 type MicrosoftGraphSelectionInput,
packages/plugins/microsoft/src/sdk/plugin.test.ts+7−6
@@ -343,12 +343,13 @@ describe("Microsoft Graph provider", () => { · 342 unmodified lines
343 ...MICROSOFT_GRAPH_DELEGATED_DEFAULT_SCOPES,343 ...MICROSOFT_GRAPH_DELEGATED_DEFAULT_SCOPES,
344 ]);344 ]);
345 345
346 // Full-graph add routes through the streaming `parsedDocument` persist346 // Full-graph add routes through the streaming compile (the path the
347 // branch (the path the real 37MB spec takes): it persists each op's347 // real 37MB spec takes): the source text is structurally split and each
348 // binding plus a `description` and writes the content-addressed defs348 // op's binding plus a `description` is persisted, alongside the
349 // blob, never re-parsing on serve. Read the operations back through the349 // content-addressed defs blob, never materializing the whole-document
350 // live serve path to prove they landed in storage AND that the serve350 // tree. Read the operations back through the live serve path to prove
351 // fast path rebuilds tools from the persisted bindings (no spec parse).351 // they landed in storage AND that the serve fast path rebuilds tools
352 // from the persisted bindings (no spec parse).
352 yield* executor.connections.create({353 yield* executor.connections.create({
353 owner: "org",354 owner: "org",
354 name: ConnectionName.make("full"),355 name: ConnectionName.make("full"),
packages/plugins/microsoft/src/sdk/plugin.ts+17−22
@@ -19,27 +19,26 @@ import { · 18 unmodified lines
19} from "@executor-js/sdk/core";19} from "@executor-js/sdk/core";
20import { describeApiKeyAuthMethod } from "@executor-js/sdk/http-auth";20import { describeApiKeyAuthMethod } from "@executor-js/sdk/http-auth";
21import {21import {
22 compileAndPersistOpenApiOperations,22 compileAndPersistOpenApiSpecStreaming,
23 compileAndPersistOpenApiSpec,
24 decodeOpenApiIntegrationConfig,23 decodeOpenApiIntegrationConfig,
25 invokeOpenApiBackedTool,24 invokeOpenApiBackedTool,
26 makeDefaultOpenapiStore,25 makeDefaultOpenapiStore,
27 normalizeOpenApiAuthInputs,26 normalizeOpenApiAuthInputs,
28 OpenApiExtractionError,27 OpenApiExtractionError,
29 OpenApiParseError,
30 resolveOpenApiBackedAnnotations,28 resolveOpenApiBackedAnnotations,
31 resolveOpenApiBackedTools,29 resolveOpenApiBackedTools,
32 type Authentication,30 type Authentication,
33 type AuthenticationInput,31 type AuthenticationInput,
34 type OpenApiPersistResult,32 type OpenApiPersistResult,
35 type OpenapiStore,33 type OpenapiStore,
36 type ParsedDocument,
37} from "@executor-js/plugin-openapi";34} from "@executor-js/plugin-openapi";
38 35
39import {36import {
40 buildMicrosoftGraphOpenApiSpec,37 buildMicrosoftGraphOpenApiSpec,
41 decodeMicrosoftGraphIntegrationConfig,38 decodeMicrosoftGraphIntegrationConfig,
39 microsoftGraphKeepPathItem,
42 type MicrosoftGraphIntegrationConfig,40 type MicrosoftGraphIntegrationConfig,
41 type MicrosoftGraphSpecBuild,
43} from "./graph";42} from "./graph";
44import {43import {
45 MICROSOFT_CLIENT_CREDENTIALS_AUTH_TEMPLATE_SLUG,44 MICROSOFT_CLIENT_CREDENTIALS_AUTH_TEMPLATE_SLUG,
@@ -127,26 +126,22 @@ const makeMicrosoftPluginExtension = ( · 81 unmodified lines
127 httpClientLayer: Layer.Layer<HttpClient.HttpClient, never, never>,126 httpClientLayer: Layer.Layer<HttpClient.HttpClient, never, never>,
128) => {127) => {
129 const persistGraphOperations = (128 const persistGraphOperations = (
130 graph: { readonly parsedDocument?: ParsedDocument; readonly specText: string },129 graph: MicrosoftGraphSpecBuild,
131 integration: string,130 integration: string,
132 specHash: string,131 specHash: string,
133 ): Effect.Effect<132 ): Effect.Effect<OpenApiPersistResult, OpenApiExtractionError | StorageFailure> =>
134 OpenApiPersistResult,133 // Stream the (full 37MB) source straight to persisted bindings + a
135 OpenApiExtractionError | OpenApiParseError | StorageFailure134 // content-addressed defs blob, never materializing the whole-document tree
136 > =>135 // that OOMs the 128MB Workers isolate. `keepPathItem` applies the Microsoft
137 graph.parsedDocument !== undefined136 // Graph scope selection per path-item during the stream; a full-graph
138 ? compileAndPersistOpenApiOperations({137 // selection returns `undefined` (keep everything).
139 doc: graph.parsedDocument,138 compileAndPersistOpenApiSpecStreaming({
140 integration,139 specText: graph.specText,
141 storage: ctx.storage,140 integration,
142 specHash,141 storage: ctx.storage,
143 })142 specHash,
144 : compileAndPersistOpenApiSpec({143 keepPathItem: microsoftGraphKeepPathItem(graph),
145 specText: graph.specText,144 });
146 integration,
147 storage: ctx.storage,
148 specHash,
149 });
150 145
151 const addGraph = (config: MicrosoftGraphConfig) =>146 const addGraph = (config: MicrosoftGraphConfig) =>
152 Effect.gen(function* () {147 Effect.gen(function* () {
packages/plugins/openapi/src/sdk/backing.ts+98−2
@@ -21,10 +21,16 @@ import { · 20 unmodified lines
21 type OpenApiIntegrationConfig,21 type OpenApiIntegrationConfig,
22} from "./config";22} from "./config";
23import { OpenApiExtractionError, OpenApiParseError } from "./errors";23import { OpenApiExtractionError, OpenApiParseError } from "./errors";
24import { buildInputSchema, extract, streamOperationBindings } from "./extract";24import {
25 buildInputSchema,
26 extract,
27 streamOperationBindings,
28 streamOperationBindingsFromStructure,
29} from "./extract";
25import { compileToolDefinitions, type ToolDefinition } from "./definitions";30import { compileToolDefinitions, type ToolDefinition } from "./definitions";
26import { annotationsForOperation, invokeWithLayer } from "./invoke";31import { annotationsForOperation, invokeWithLayer } from "./invoke";
27import { parse, type ParsedDocument } from "./parse";32import { parse, type ParsedDocument } from "./parse";
33import { parseEntry, structuralSplit, type KeepPathItem, type SpecStructure } from "./split";
28import { type OpenapiStore, type StoredOperation } from "./store";34import { type OpenapiStore, type StoredOperation } from "./store";
29import { OperationBinding } from "./types";35import { OperationBinding } from "./types";
30 36
@@ -282,6 +288,32 @@ export const buildDefsJson = (doc: ParsedDocument): string => { · 251 unmodified lines
282 return `${json}}`;288 return `${json}}`;
283};289};
284 290
291/**
292 * Streaming twin of `buildDefsJson`: serialize the content-addressed defs blob
293 * from a `SpecStructure` by parsing each `components.schemas` entry in isolation
294 * (indent-4 range) rather than from a whole-document parse. Used by the fully
295 * streaming add path so the 37MB Microsoft Graph spec never builds its
296 * ~300MB tree. The blob carries *all* source schemas (it is shared across every
297 * tenant/selection on the same spec hash); extra unreferenced `$defs` are
298 * harmless. Like `buildDefsJson`, normalizing + stringifying per entry keeps the
299 * whole normalized tree from being co-resident with any parsed schema, and the
300 * ConsString accumulation avoids the join-doubling of an array build.
301 */
302export const buildDefsJsonStreaming = (structure: SpecStructure): string => {
303 let json = "{";
304 let first = true;
305 for (const range of structure.schemas) {
306 const entry = parseEntry(structure.text, range, 4);
307 if (!entry) continue;
308 const [name, schema] = entry;
309 const serialized = JSON.stringify(normalizeOpenApiRefs(schema));
310 if (serialized === undefined) continue;
311 json += `${first ? "" : ","}${JSON.stringify(name)}:${serialized}`;
312 first = false;
313 }
314 return `${json}}`;
315};
316
285const DefsJson = Schema.Record(Schema.String, Schema.Unknown);317const DefsJson = Schema.Record(Schema.String, Schema.Unknown);
286/** Decode the content-addressed defs blob back into the shared `definitions`318/** Decode the content-addressed defs blob back into the shared `definitions`
287 * map. Returns `None` on a corrupt/non-object blob so the serve path falls319 * map. Returns `None` on a corrupt/non-object blob so the serve path falls
@@ -402,6 +434,65 @@ export const compileAndPersistOpenApiSpec = ({ · 114 unmodified lines
402 });434 });
403 });435 });
404 436
437/**
438 * Fully streaming add/update path: compile + persist operation bindings (and the
439 * content-addressed defs blob) straight from spec *text*, without ever parsing
440 * the whole document. The text is structurally split, then each path-item and
441 * each schema is parsed in isolation and discarded, so peak memory stays near
442 * one item rather than the ~300MB whole-tree parse that OOMs a 128MB Workers
443 * isolate on the 37MB Microsoft Graph spec.
444 *
445 * There is deliberately no whole-parse fallback: a spec that does not present
446 * the streamable block-YAML profile (no top-level `paths:` block) is a hard
447 * `OpenApiExtractionError`, because the fallback is exactly the OOM this path
448 * exists to avoid. `keepPathItem` optionally filters/trims path-items (the
449 * Microsoft Graph scope selection), so the same primitive serves a full-spec
450 * compile and a selection identically.
451 */
452export const compileAndPersistOpenApiSpecStreaming = ({
453 specText,
454 integration,
455 storage,
456 specHash,
457 chunkSize,
458 keepPathItem,
459}: {
460 readonly specText: string;
461 readonly integration: string;
462 readonly storage: OpenapiStore;
463 readonly specHash?: string;
464 readonly chunkSize?: number;
465 readonly keepPathItem?: KeepPathItem;
466}): Effect.Effect<OpenApiPersistResult, OpenApiExtractionError | StorageFailure> =>
467 Effect.gen(function* () {
468 const structure = structuralSplit(specText);
469 if (!structure) {
470 return yield* new OpenApiExtractionError({
471 message:
472 "OpenAPI spec is not in the streamable block-YAML profile (no top-level `paths:` block); cannot stream-compile a spec this large in-band.",
473 });
474 }
475 yield* storage.removeOperations(integration);
476 const result = yield* streamOperationBindingsFromStructure(
477 structure,
478 { chunkSize: chunkSize ?? 500, keepPathItem },
479 (chunk) =>
480 storage.appendOperations(
481 integration,
482 chunk.map((item) => ({
483 integration,
484 toolName: item.toolName,
485 binding: item.binding,
486 description: item.description,
487 })),
488 ),
489 );
490 if (specHash != null) {
491 yield* storage.putDefs(specHash, buildDefsJsonStreaming(structure));
492 }
493 return result;
494 });
495
405export const loadOpenApiSpecText = (496export const loadOpenApiSpecText = (
406 storage: OpenapiStore,497 storage: OpenapiStore,
407 config: OpenApiIntegrationConfig,498 config: OpenApiIntegrationConfig,
@@ -464,7 +555,12 @@ export const invokeOpenApiBackedTool = (input: { · 56 unmodified lines
464 const config = decodeOpenApiIntegrationConfig(input.credential.config);555 const config = decodeOpenApiIntegrationConfig(input.credential.config);
465 556
466 let binding = (yield* input.ctx.storage.getOperation(integration, input.toolRow.name))?.binding;557 let binding = (yield* input.ctx.storage.getOperation(integration, input.toolRow.name))?.binding;
467 if ((!binding || Option.isNone(binding.responseBody)) && config) {558 // Only re-parse when the binding is entirely absent (a legacy row predating
559 // persisted bindings). A present binding is authoritative even if it has no
560 // responseBody: the persisted spec is now the *full* source (37MB for
561 // Microsoft Graph), so re-parsing it here to "enrich" a binding would OOM
562 // the isolate. A genuinely body-less operation must serve from its binding.
563 if (!binding && config) {
468 const specText = yield* loadOpenApiSpecText(input.ctx.storage, config).pipe(564 const specText = yield* loadOpenApiSpecText(input.ctx.storage, config).pipe(
469 Effect.catch(() => Effect.succeed(null)),565 Effect.catch(() => Effect.succeed(null)),
470 );566 );
packages/plugins/openapi/src/sdk/extract.ts+140−1
@@ -1,8 +1,16 @@
1import { Effect, Option } from "effect";1import { Effect, Option } from "effect";
2 2
3import { planToolPaths, type OperationPathInput } from "./definitions";3import { planToolPaths, type OperationPathInput, type PlannedToolPath } from "./definitions";
4import { OpenApiExtractionError } from "./errors";4import { OpenApiExtractionError } from "./errors";
5import type { ParsedDocument } from "./parse";5import type { ParsedDocument } from "./parse";
6import {
7 parseEntry,
8 parseHead,
9 parseSmallComponents,
10 type ByteRange,
11 type KeepPathItem,
12 type SpecStructure,
13} from "./split";
6import {14import {
7 declaredContents,15 declaredContents,
8 DocResolver,16 DocResolver,
@@ -622,3 +630,134 @@ export const streamOperationBindings = <E, R>( · 613 unmodified lines
622 630
623 return { toolCount: plans.length, toolNames: plans.map((plan) => plan.toolPath) };631 return { toolCount: plans.length, toolNames: plans.map((plan) => plan.toolPath) };
624 }).pipe(Effect.withSpan("OpenApi.streamOperationBindings"));632 }).pipe(Effect.withSpan("OpenApi.streamOperationBindings"));
633
634const isPathItemValue = (value: unknown): value is Record<string, unknown> =>
635 value !== null && typeof value === "object" && !Array.isArray(value);
636
637/**
638 * Stream invocation bindings straight from a `SpecStructure` (the structural
639 * split of a large spec) without ever materializing the whole-document tree.
640 *
641 * This is the fully-streaming compile path: it never parses the spec whole.
642 * Each path-item is parsed in isolation twice (pass 1 for light tool-path
643 * planning metadata, pass 2 to build the heavy binding) and discarded, so peak
644 * memory stays near the size of a single path-item plus the raw text, even for
645 * the 37MB / 16.5k-operation Microsoft Graph spec that OOMs a whole-tree parse.
646 * Re-parsing in pass 2 (rather than holding pass-1 tree references) is the
647 * deliberate CPU-for-memory trade that keeps peak at one-path-item level.
648 *
649 * `keepPathItem`, when given, filters (and may trim) each path-item, so the same
650 * primitive serves both a full-spec compile (no filter) and a selection (e.g.
651 * the Microsoft Graph scope filter) with identical streaming guarantees. It is
652 * applied identically in both passes so the per-operation index stays aligned.
653 *
654 * Schemas are never resolved here: parameter / requestBody / response component
655 * `$ref`s resolve against the small (schema-free) components built from the
656 * structure; `#/components/schemas/X` refs stay as strings in the binding and
657 * are normalized + served from the content-addressed defs blob. Returns the
658 * resolved tool names in sorted order, matching `compileToolDefinitions`.
659 */
660export const streamOperationBindingsFromStructure = <E, R>(
661 structure: SpecStructure,
662 options: { readonly chunkSize: number; readonly keepPathItem?: KeepPathItem },
663 onChunk: (chunk: readonly OperationBindingChunk[]) => Effect.Effect<void, E, R>,
664): Effect.Effect<{ readonly toolCount: number; readonly toolNames: readonly string[] }, E, R> =>
665 Effect.gen(function* () {
666 const { chunkSize, keepPathItem } = options;
667
668 // Parse one path-item range to its kept (optionally trimmed) value, applying
669 // `keepPathItem` identically in both passes so the operation index aligns.
670 const keptPathItem = (range: ByteRange): readonly [string, PathItemObject] | null => {
671 const entry = parseEntry(structure.text, range, 2);
672 if (!entry) return null;
673 const [path, rawValue] = entry;
674 if (!isPathItemValue(rawValue)) return null;
675 if (!keepPathItem) return [path, rawValue as PathItemObject];
676 const kept = keepPathItem(path, rawValue);
677 return kept ? [path, kept as PathItemObject] : null;
678 };
679
680 // Pass 1 (light): collect schema-free tool-path planning metadata in
681 // document order. No bindings, no schemas; one path-item resident at a time.
682 const inputs: OperationPathInput[] = [];
683 for (const range of structure.pathItems) {
684 const kept = keptPathItem(range);
685 if (!kept) continue;
686 const [path, pathItem] = kept;
687 for (const method of HTTP_METHODS) {
688 const operation = pathItem[method];
689 if (!operation) continue;
690 const resolvedPathTemplate = explicitPathTemplate(operation) ?? path;
691 const tags = (operation.tags ?? []).filter((t) => t.trim().length > 0);
692 inputs.push({
693 operationId: deriveOperationId(method, path, operation),
694 explicitToolPath: explicitToolPath(operation),
695 method,
696 pathTemplate: resolvedPathTemplate,
697 tag0: tags[0],
698 });
699 }
700 }
701
702 // Global, schema-free collision resolution + sort. `plan.operationIndex`
703 // indexes back into `inputs` (document order), so a flat array recovers each
704 // operation's assigned tool path during the document-order pass 2.
705 const plans = planToolPaths(inputs);
706 const planByOpIndex: (PlannedToolPath | undefined)[] = new Array(inputs.length);
707 for (const plan of plans) planByOpIndex[plan.operationIndex] = plan;
708
709 // Pass 2 (heavy, streamed): re-parse each path-item in the same document
710 // order, build a binding per operation, flush a chunk once it fills, then
711 // drop it. The resolver is schema-free (small components only); schema refs
712 // stay as `$ref` strings in the bindings.
713 // oxlint-disable-next-line executor/no-double-cast -- boundary: parseHead/parseSmallComponents return Record<string, unknown>, which does not structurally match the OpenAPIV3 Document union; the schema-free resolver doc (head + small components, empty paths) is only read for .servers (extractServers) and .components ($ref resolution).
714 const resolverDoc = {
715 ...parseHead(structure),
716 paths: {},
717 components: parseSmallComponents(structure),
718 } as unknown as ParsedDocument;
719 const r = new DocResolver(resolverDoc);
720 const docServers = extractServers(resolverDoc);
721
722 let opIndex = 0;
723 let chunk: OperationBindingChunk[] = [];
724 for (const range of structure.pathItems) {
725 const kept = keptPathItem(range);
726 if (!kept) continue;
727 const [path, pathItem] = kept;
728 for (const method of HTTP_METHODS) {
729 const operation = pathItem[method];
730 if (!operation) continue;
731 const plan = planByOpIndex[opIndex];
732 opIndex += 1;
733 if (!plan) continue;
734 const resolvedPathTemplate = explicitPathTemplate(operation) ?? path;
735 const parameters = extractParameters(pathItem, operation, r);
736 const requestBody = extractRequestBody(operation, r);
737 const responseBody = extractResponseBody(operation, r);
738 const servers = operationServers(pathItem, operation, docServers);
739 chunk.push({
740 toolName: plan.toolPath,
741 description:
742 operation.description ??
743 operation.summary ??
744 `${method.toUpperCase()} ${resolvedPathTemplate}`,
745 binding: OperationBinding.make({
746 method,
747 servers,
748 pathTemplate: resolvedPathTemplate,
749 parameters,
750 requestBody: Option.fromNullishOr(requestBody),
751 responseBody: Option.fromNullishOr(responseBody),
752 }),
753 });
754 if (chunk.length >= chunkSize) {
755 yield* onChunk(chunk);
756 chunk = [];
757 }
758 }
759 }
760 if (chunk.length > 0) yield* onChunk(chunk);
761
762 return { toolCount: plans.length, toolNames: plans.map((plan) => plan.toolPath) };
763 }).pipe(Effect.withSpan("OpenApi.streamOperationBindingsFromStructure"));
packages/plugins/openapi/src/sdk/index.ts+15−1
@@ -1,9 +1,23 @@
1export { parse, resolveSpecText, fetchSpecText } from "./parse";1export { parse, resolveSpecText, fetchSpecText } from "./parse";
2export { extract } from "./extract";2export { extract, streamOperationBindingsFromStructure } from "./extract";
3export {
4 structuralSplit,
5 isStreamableSpec,
6 indexSchemas,
7 collectReferencedSchemas,
8 parseEntry,
9 parseHead,
10 parseSmallComponents,
11 type SpecStructure,
12 type ByteRange,
13 type KeepPathItem,
14} from "./split";
3export { invoke, invokeWithLayer, annotationsForOperation } from "./invoke";15export { invoke, invokeWithLayer, annotationsForOperation } from "./invoke";
4export {16export {
17 buildDefsJsonStreaming,
5 compileAndPersistOpenApiOperations,18 compileAndPersistOpenApiOperations,
6 compileAndPersistOpenApiSpec,19 compileAndPersistOpenApiSpec,
20 compileAndPersistOpenApiSpecStreaming,
7 compileOpenApiDocument,21 compileOpenApiDocument,
8 compileOpenApiSpec,22 compileOpenApiSpec,
9 extractOpenApiUpstreamMessage,23 extractOpenApiUpstreamMessage,
packages/plugins/openapi/src/sdk/split.ts+387−0
@@ -0,0 +1,387 @@
1/**
2 * Structural splitter for large OpenAPI documents written in clean block YAML.
3 *
4 * The whole-document parse of a 37MB spec (Microsoft Graph: 16.5k operations,
5 * 8.2k schemas) builds a ~300MB JS tree that OOMs the 128MB Cloudflare Workers
6 * isolate. This splitter avoids ever holding that tree: it scans the text once,
7 * recording the byte range of each top-level key, each path-item (the indent-2
8 * entries under `paths:`), and each schema (the indent-4 entries under
9 * `components.schemas:`). The streaming compile then slices one item at a time,
10 * de-indents it back to column 0, hands the isolated fragment to a real YAML
11 * parser, and discards the result before moving on. Peak memory stays near the
12 * size of the largest single item plus the raw text, not the parsed whole.
13 *
14 * This is not a YAML reimplementation: it extracts safe byte ranges from a
15 * constrained document shape, then defers every actual parse to `js-yaml`. It
16 * is only valid for the block-YAML profile `isStreamableSpec` accepts (2-space
17 * block maps, top-level keys at column 0, no anchors/aliases/merge keys). Block
18 * scalars (`|` / `>`) in descriptions are tracked so their indented content is
19 * never mistaken for structure.
20 */
21
22import { JSON_SCHEMA, load as parseYamlDocument } from "js-yaml";
23
24export interface ByteRange {
25 readonly start: number;
26 readonly end: number;
27}
28
29export interface SpecStructure {
30 /** Raw spec text. Every range below indexes into this string. */
31 readonly text: string;
32 /** Byte ranges of the top-level keys that are not `paths` / `components`
33 * (openapi, info, servers, tags, security, ...). Concatenated and parsed as
34 * one small document for the head (servers, info). */
35 readonly headRanges: readonly ByteRange[];
36 /** One range per path-item: the indent-2 entries under `paths:`. */
37 readonly pathItems: readonly ByteRange[];
38 /** One range per schema entry: the indent-4 entries under
39 * `components.schemas:`. */
40 readonly schemas: readonly ByteRange[];
41 /** Ranges of the indent-2 `components` subkeys we keep whole because they are
42 * small and may be `$ref`'d by a kept operation (parameters / requestBodies /
43 * responses / headers / links / securitySchemes). Excludes the huge `schemas`
44 * (streamed and pruned separately) and `examples` (never referenced by a
45 * binding). */
46 readonly smallComponentRanges: readonly ByteRange[];
47}
48
49const SPACE = 32;
50const HASH = 35;
51const DASH = 45;
52
53/** `components` subkeys kept whole during a reduce: small, and a kept operation
54 * may `$ref` into any of them. `schemas` is streamed/pruned separately and
55 * `examples` is dropped (large, never referenced by a binding). */
56const SMALL_COMPONENT_SECTIONS = new Set([
57 "parameters",
58 "requestBodies",
59 "responses",
60 "headers",
61 "links",
62 "securitySchemes",
63]);
64
65/** Count of leading spaces on the line starting at `lineStart`. */
66const indentOf = (text: string, lineStart: number, lineEnd: number): number => {
67 let i = lineStart;
68 while (i < lineEnd && text.charCodeAt(i) === SPACE) i++;
69 return i - lineStart;
70};
71
72/** True for a blank (whitespace-only) or comment line. */
73const isBlankOrComment = (text: string, contentStart: number, lineEnd: number): boolean =>
74 contentStart >= lineEnd || text.charCodeAt(contentStart) === HASH;
75
76/**
77 * True when the line opens a block scalar (`key: |`, `key: >-`, etc.), meaning
78 * every following more-indented line is literal content, not structure.
79 */
80const opensBlockScalar = (line: string): boolean =>
81 /:\s*[|>][+-]?\d*\s*(#.*)?$/.test(line) || /^\s*[|>][+-]?\d*\s*(#.*)?$/.test(line);
82
83interface LineCursor {
84 readonly lineStart: number;
85 readonly lineEnd: number;
86 readonly nextStart: number;
87 readonly indent: number;
88 readonly contentStart: number;
89}
90
91/** Advance to the next significant line at offset `from`, returning null at EOF.
92 * Blank lines, comments, and block-scalar content (lines indented deeper than
93 * the scalar key, tracked by the caller) are still returned; the caller skips
94 * them. */
95const lineAt = (text: string, from: number, limit: number): LineCursor | null => {
96 if (from >= limit) return null;
97 let lineEnd = text.indexOf("\n", from);
98 if (lineEnd === -1 || lineEnd > limit) lineEnd = limit;
99 const indent = indentOf(text, from, lineEnd);
100 const contentStart = from + indent;
101 return {
102 lineStart: from,
103 lineEnd,
104 nextStart: lineEnd + 1,
105 indent,
106 contentStart,
107 };
108};
109
110/**
111 * Find the start offset of every key line at exactly `indent` within
112 * `[start, end)`, skipping blank/comment lines, deeper child lines, and
113 * block-scalar content. Sequence items (`- `) at the target indent are not
114 * treated as keys. Returns offsets in document order.
115 */
116const keyLineStartsAtIndent = (
117 text: string,
118 start: number,
119 end: number,
120 indent: number,
121): number[] => {
122 const starts: number[] = [];
123 let blockScalarIndent = -1;
124 let pos = start;
125 while (pos < end) {
126 const line = lineAt(text, pos, end);
127 if (!line) break;
128 pos = line.nextStart;
129 if (isBlankOrComment(text, line.contentStart, line.lineEnd)) continue;
130 if (blockScalarIndent >= 0) {
131 if (line.indent > blockScalarIndent) continue;
132 blockScalarIndent = -1;
133 }
134 if (line.indent === indent && text.charCodeAt(line.contentStart) !== DASH) {
135 starts.push(line.lineStart);
136 }
137 if (line.indent <= indent && opensBlockScalar(text.slice(line.contentStart, line.lineEnd))) {
138 blockScalarIndent = line.indent;
139 }
140 }
141 return starts;
142};
143
144/** Contiguous ranges from a sorted list of block-start offsets within
145 * `[blockStart, blockEnd)`: each range runs to the next start (or blockEnd). */
146const rangesFromStarts = (starts: readonly number[], blockEnd: number): ByteRange[] =>
147 starts.map((s, i) => ({ start: s, end: i + 1 < starts.length ? starts[i + 1]! : blockEnd }));
148
149/** The simple (unquoted) key name on a key line, e.g. `paths`, `schemas`. */
150const keyNameAt = (text: string, lineStart: number, lineEnd: number): string => {
151 const indent = indentOf(text, lineStart, lineEnd);
152 const colon = text.indexOf(":", lineStart + indent);
153 const keyEnd = colon === -1 || colon > lineEnd ? lineEnd : colon;
154 return text.slice(lineStart + indent, keyEnd).trim();
155};
156
157/**
158 * Scan a document into its structural ranges. Pure and synchronous; never
159 * parses. Returns null when the document does not present the expected shape
160 * (no `paths:` block), so the caller can fall back to a whole-document parse.
161 */
162export const structuralSplit = (text: string): SpecStructure | null => {
163 const len = text.length;
164
165 // Top-level keys (column 0). Record each key's start + name, respecting
166 // block scalars whose content could sit at column >= 1.
167 const topStarts: number[] = [];
168 const topNames: string[] = [];
169 {
170 let blockScalarIndent = -1;
171 let pos = 0;
172 while (pos < len) {
173 const line = lineAt(text, pos, len);
174 if (!line) break;
175 pos = line.nextStart;
176 if (isBlankOrComment(text, line.contentStart, line.lineEnd)) continue;
177 if (blockScalarIndent >= 0) {
178 if (line.indent > blockScalarIndent) continue;
179 blockScalarIndent = -1;
180 }
181 if (line.indent === 0) {
182 topStarts.push(line.lineStart);
183 topNames.push(keyNameAt(text, line.lineStart, line.lineEnd));
184 }
185 if (opensBlockScalar(text.slice(line.contentStart, line.lineEnd))) {
186 blockScalarIndent = line.indent;
187 }
188 }
189 }
190
191 const topRanges = rangesFromStarts(topStarts, len);
192 const pathsIdx = topNames.indexOf("paths");
193 if (pathsIdx === -1) return null;
194 const componentsIdx = topNames.indexOf("components");
195
196 const headRanges: ByteRange[] = [];
197 for (let i = 0; i < topRanges.length; i++) {
198 if (i === pathsIdx || i === componentsIdx) continue;
199 headRanges.push(topRanges[i]!);
200 }
201
202 // Path-items: indent-2 keys inside the `paths:` block (after its key line).
203 const pathsRange = topRanges[pathsIdx]!;
204 const pathsBodyStart =
205 lineAt(text, pathsRange.start, pathsRange.end)?.nextStart ?? pathsRange.end;
206 const pathItems = rangesFromStarts(
207 keyLineStartsAtIndent(text, pathsBodyStart, pathsRange.end, 2),
208 pathsRange.end,
209 );
210
211 const schemas: ByteRange[] = [];
212 const smallComponentRanges: ByteRange[] = [];
213 if (componentsIdx !== -1) {
214 const componentsRange = topRanges[componentsIdx]!;
215 const componentsBodyStart =
216 lineAt(text, componentsRange.start, componentsRange.end)?.nextStart ?? componentsRange.end;
217 const subStarts = keyLineStartsAtIndent(text, componentsBodyStart, componentsRange.end, 2);
218 const subRanges = rangesFromStarts(subStarts, componentsRange.end);
219 for (const range of subRanges) {
220 const name = keyNameAt(text, range.start, lineAt(text, range.start, range.end)!.lineEnd);
221 if (name === "schemas") {
222 const bodyStart = lineAt(text, range.start, range.end)?.nextStart ?? range.end;
223 for (const s of rangesFromStarts(
224 keyLineStartsAtIndent(text, bodyStart, range.end, 4),
225 range.end,
226 )) {
227 schemas.push(s);
228 }
229 } else if (SMALL_COMPONENT_SECTIONS.has(name)) {
230 smallComponentRanges.push(range);
231 }
232 // `examples` and any other subkey are intentionally dropped: no binding
233 // field references them, and they can be large.
234 }
235 }
236
237 return { text, headRanges, pathItems, schemas, smallComponentRanges };
238};
239
240/** Strip exactly `indent` leading spaces from every line of `fragment`, lifting
241 * an indent-N block back to column 0 so it parses as a standalone document. */
242const dedent = (fragment: string, indent: number): string =>
243 indent === 0 ? fragment : fragment.replace(new RegExp(`^ {1,${indent}}`, "gm"), "");
244
245const parseYaml = (text: string): unknown =>
246 parseYamlDocument(text, { json: true, schema: JSON_SCHEMA });
247
248/**
249 * Parse a single indent-N entry range (a path-item or a schema) in isolation.
250 * Returns `[name, value]` where `name` is the entry's key and `value` its
251 * parsed body, or null when the fragment does not parse to a single-key map.
252 */
253export const parseEntry = (
254 text: string,
255 range: ByteRange,
256 indent: number,
257): readonly [string, unknown] | null => {
258 const parsed = parseYaml(dedent(text.slice(range.start, range.end), indent));
259 if (parsed === null || typeof parsed !== "object" || Array.isArray(parsed)) return null;
260 const entries = Object.entries(parsed as Record<string, unknown>);
261 if (entries.length !== 1) return null;
262 return entries[0]!;
263};
264
265/** Parse the concatenation of the head ranges into the document head (openapi,
266 * info, servers, tags). Small; safe to materialize whole. */
267export const parseHead = (structure: SpecStructure): Record<string, unknown> => {
268 const text = structure.headRanges.map((r) => structure.text.slice(r.start, r.end)).join("");
269 const parsed = parseYaml(text);
270 return parsed && typeof parsed === "object" && !Array.isArray(parsed)
271 ? (parsed as Record<string, unknown>)
272 : {};
273};
274
275/** Parse the small component subkeys (parameters / requestBodies / responses)
276 * into a `components`-shaped object for `$ref` resolution. */
277export const parseSmallComponents = (structure: SpecStructure): Record<string, unknown> => {
278 if (structure.smallComponentRanges.length === 0) return {};
279 const body = structure.smallComponentRanges
280 .map((r) => structure.text.slice(r.start, r.end))
281 .join("");
282 const parsed = parseYaml(`components:\n${body}`);
283 const components =
284 parsed && typeof parsed === "object" ? (parsed as Record<string, unknown>).components : null;
285 return components && typeof components === "object"
286 ? (components as Record<string, unknown>)
287 : {};
288};
289
290/**
291 * Accept only the block-YAML profile the splitter can safely slice: no tabs, no
292 * anchors/aliases, no merge keys, and a `paths:` block present. Block scalars
293 * are allowed (tracked during the scan). Conservative by design: a false
294 * negative just routes a spec through the whole-document parse.
295 */
296export const isStreamableSpec = (text: string): boolean => {
297 if (text.indexOf("\t") !== -1) return false;
298 // Anchors/aliases/merge keys: cheap substring rejects before structural work.
299 if (/(^|\s)[&*][A-Za-z0-9_]/m.test(text)) return false;
300 if (/<<\s*:/.test(text)) return false;
301 return /^paths:/m.test(text);
302};
303
304const SCHEMA_REF_PREFIX = "#/components/schemas/";
305
306/** Map each `components.schemas` entry name to its byte range, reading only the
307 * key line (never parsing the schema body). The schema name is the raw YAML
308 * key, which matches the trailing segment of a `#/components/schemas/<name>`
309 * reference. */
310export const indexSchemas = (structure: SpecStructure): ReadonlyMap<string, ByteRange> => {
311 const index = new Map<string, ByteRange>();
312 for (const range of structure.schemas) {
313 const line = lineAt(structure.text, range.start, range.end);
314 if (!line) continue;
315 const name = keyNameAt(structure.text, range.start, line.lineEnd);
316 if (name) index.set(name, range);
317 }
318 return index;
319};
320
321/** Decode a `#/components/schemas/<segment>` name segment. Schema names are raw
322 * YAML keys, so only JSON-pointer tilde escaping can appear (no `%`-encoding
323 * in the wild specs we target); decode `~1`/`~0` and leave the rest verbatim. */
324const decodeSchemaRefName = (segment: string): string =>
325 segment.replace(/~1/g, "/").replace(/~0/g, "~");
326
327/** Collect the names of every `#/components/schemas/X` reference reachable in
328 * `value`, without resolving them. */
329const collectSchemaRefNames = (value: unknown, into: Set<string>): void => {
330 if (typeof value === "string") {
331 if (value.startsWith(SCHEMA_REF_PREFIX)) {
332 const name = decodeSchemaRefName(value.slice(SCHEMA_REF_PREFIX.length));
333 if (name.length > 0) into.add(name);
334 }
335 return;
336 }
337 if (Array.isArray(value)) {
338 for (const item of value) collectSchemaRefNames(item, into);
339 return;
340 }
341 if (value !== null && typeof value === "object") {
342 for (const item of Object.values(value as Record<string, unknown>)) {
343 collectSchemaRefNames(item, into);
344 }
345 }
346};
347
348/**
349 * Resolve the transitive closure of schemas referenced by `roots`, parsing each
350 * referenced schema once from its byte range (BFS over `$ref`s). Schemas not
351 * reachable from `roots` are never parsed, so peak memory tracks the kept
352 * subset rather than the full `components.schemas` map.
353 */
354export const collectReferencedSchemas = (
355 structure: SpecStructure,
356 index: ReadonlyMap<string, ByteRange>,
357 roots: readonly unknown[],
358): Record<string, unknown> => {
359 const result: Record<string, unknown> = {};
360 const wanted = new Set<string>();
361 for (const root of roots) collectSchemaRefNames(root, wanted);
362
363 const queue = [...wanted];
364 for (let i = 0; i < queue.length; i += 1) {
365 const name = queue[i]!;
366 if (Object.prototype.hasOwnProperty.call(result, name)) continue;
367 const range = index.get(name);
368 if (!range) continue;
369 const entry = parseEntry(structure.text, range, 4);
370 if (!entry) continue;
371 result[name] = entry[1];
372 const next = new Set<string>();
373 collectSchemaRefNames(entry[1], next);
374 for (const ref of next) {
375 if (!Object.prototype.hasOwnProperty.call(result, ref)) queue.push(ref);
376 }
377 }
378 return result;
379};
380
381/** A path-item filter for the streaming compile: given a parsed path-item,
382 * return the (possibly trimmed) value to keep, or null to drop the path
383 * entirely. Applied per path-item by `streamOperationBindingsFromStructure`. */
384export type KeepPathItem = (
385 path: string,
386 pathItem: Record<string, unknown>,
387) => Record<string, unknown> | null;