Move source auth handling into plugins

Rhys Sullivan committed 7 months ago

Commit db9b84d1 parent 02d549c15 files · +98 −125Browse files

Unified · Split

15 files

apps/docs/developer/core-model.mdx +12 −7
apps/docs/developer/credentials-and-auth.mdx +14 −11
apps/docs/developer/extending-executor.mdx +10 −3
apps/docs/developer/mcp.mdx +2 −2
apps/docs/developer/overview.mdx +4 −4
bun.lock +1 −0
packages/platform/sdk/src/runtime/index.ts +0 −1
packages/platform/sdk/src/runtime/sources/source-credential-interactions.ts +0 −34
packages/sources/core/src/shared.ts +0 −20
plugins/google-discovery/sdk/index.ts +1 −3
plugins/graphql/sdk/index.ts +1 −3
plugins/openapi/react/index.tsx +36 −14
plugins/openapi/sdk/index.ts +13 −21
plugins/openapi/shared/index.ts +3 −2
plugins/openapi/shared/package.json +1 −0

@@ -57,17 +57,17 @@ They answer: · 56 unmodified lines
57	57
58	58	The operation layer exists because raw upstream documents are not a good direct query surface for search, policy, and invocation lookup.
59	59
60		−### Credential
	60	+### Secret material
61	61
62		−Credentials are private auth bindings.
	62	+Secret material is private auth state.
63	63
64		−They answer:
	64	+It answers:
65	65
66		−- who can authenticate for this binding?
67		−- for which slot?
68		−- using what auth kind and secret refs?
	66	+- which private values exist for this workspace actor?
	67	+- which source bindings refer to them?
	68	+- which plugin-owned auth configs use those secret refs?
69	69
70		−Credentials are actor-scoped by default because two users in the same workspace may connect different identities to the same source.
	70	+Secret material is actor-scoped by default because two users in the same workspace may connect different identities to the same source.
71	71
72	72	## Why the split matters
73	73
@@ -115,6 +115,11 @@ Adapters are more specific: · 41 unmodified lines
115	115
116	116	The important distinction is that execution now belongs to adapters. The shared catalog stays broad and stable, while connect/auth/invoke behavior is described through adapter strategies instead of one top-level runtime family enum.
117	117
	118	+The same is true for auth behavior:
	119	+
	120	+- core owns secret storage and private state plumbing
	121	+- plugins own auth schemas, request shaping, OAuth flows, and refresh logic
	122	+
118	123	## Why this is the right shape
119	124
120	125	The current model is designed so that new support usually means:

apps/docs/developer/credentials-and-auth.mdx+14−11

@@ -5,11 +5,14 @@ description: "How executor models auth, interactive flows, and actor-scoped cred · 4 unmodified lines
5	5
6	6	Auth in executor is not treated as a side detail on the source row.
7	7
8		−It is modeled as its own layer because auth has different ownership and lifecycle semantics from source bindings and recipe materialization.
	8	+The important boundary is:
	9	+
	10	+- core owns secret material storage, actor scoping, and private state plumbing
	11	+- plugins own auth schemas and auth behavior for each source kind
9	12
10	13	## Auth ownership
11	14
12		−Credentials are scoped to:
	15	+Secret material is scoped to:
13	16
14	17	- workspace
15	18	- source binding
@@ -20,7 +23,7 @@ This is important because different users in the same workspace may connect diff · 4 unmodified lines
20	23
21	24	## Auth slots
22	25
23		−Executor distinguishes where auth is used.
	26	+Plugins may distinguish where auth is used.
24	27
25	28	Important current slots:
26	29
@@ -33,7 +36,7 @@ Why this matters: · 6 unmodified lines
33	36	- some need auth only to fetch or generate the materialization
34	37	- some may eventually need setup/bootstrap-specific auth
35	38
36		−This is why source auth is more than a single token field.
	39	+This is why plugin-owned source auth is more than a single token field.
37	40
38	41	## Static credentials
39	42
@@ -45,11 +48,11 @@ Examples: · 5 unmodified lines
45	48	- OAuth access token
46	49	- OAuth refresh token
47	50
48		−The control plane stores secret refs, not raw secret values inline on the source or credential rows.
	51	+The control plane stores secret refs, not raw secret values inline on the source or plugin-owned auth configs.
49	52
50	53	## Interactive auth sessions
51	54
52		−Interactive auth, such as OAuth flows, is modeled separately from stored credentials.
	55	+Interactive auth, such as OAuth flows, is modeled separately from stored secret material.
53	56
54	57	That separation exists because:
55	58
@@ -57,7 +60,7 @@ That separation exists because: · 1 unmodified line
57	60	- it may be tied to an interaction or execution
58	61	- it is not the same thing as a resolved credential
59	62
60		−This is why executor has auth sessions and credentials as separate concepts.
	63	+This is why executor keeps auth sessions separate from stored secret material.
61	64
62	65	## Shareable config vs private state
63	66
@@ -65,7 +68,7 @@ Project config under `.executor/executor.jsonc` is intended to stay portable and · 1 unmodified line
65	68
66	69	Private auth state does not belong in the workspace tree:
67	70
68		−- actor-scoped credentials
	71	+- actor-scoped secret material
69	72	- OAuth sessions
70	73	- refresh leases
71	74	- secret material
@@ -97,17 +100,17 @@ The likely next auth-layer primitive is an auth provider or auth material resolv · 25 unmodified lines
97	100	- produces runtime auth material on demand
98	101	- caches it with expiry metadata
99	102
100		−That is broader than sources and should be reusable by future source adapters, tool providers, and code bundles.
	103	+That is broader than sources and should be reusable by future plugins, tool providers, and code bundles.
101	104
102	105	## Why auth is separated from recipes
103	106
104	107	Recipes are shareable integration definitions.
105		−Credentials are private installation-specific state.
	108	+Secret material is private installation-specific state.
106	109
107	110	That separation is important both for correctness and for shareability:
108	111
109	112	- a recipe can be portable
110	113	- a source binding can be local
111		−- credentials stay private
	114	+- secret material stays private
112	115
113	116	This is the main reason auth is not modeled as part of the shareable recipe layer.

apps/docs/developer/extending-executor.mdx+10−3

@@ -36,6 +36,12 @@ The happy path for a new integration is: · 35 unmodified lines
36	36
37	37	The adapter should absorb provider-specific logic so the core does not have to.
38	38
	39	+That includes auth behavior:
	40	+
	41	+- plugin-owned auth config
	42	+- plugin-owned OAuth flows and refresh logic
	43	+- core-owned secret material services
	44	+
39	45	## What a new adapter should not do
40	46
41	47	Avoid these patterns:
@@ -54,7 +60,8 @@ A good new adapter should make the core system feel boring. · 12 unmodified lines
54	60	That means:
55	61
56	62	- persistence still looks like recipe/revision/operations
57		−- auth still flows through the same credential/session primitives
	63	+- auth still uses the same secret material and private-state plumbing
	64	+- auth behavior itself stays plugin-owned
58	65	- discovery and inspection still work through the same catalog ideas
59	66	- execution still delegates through the same tool and adapter boundaries
60	67
@@ -69,7 +76,7 @@ The current architecture is well-suited to future additions like: · 8 unmodified lines
69	76	- snippet bundles
70	77	- richer auth material providers
71	78
72		−Those should mostly be adapter and auth-layer work, not another persistence redesign.
	79	+Those should mostly be adapter and secret-material-layer work, not another persistence redesign.
73	80
74	81	## Why this is the best setup
75	82
@@ -80,4 +87,4 @@ It is trying to keep the codebase in a state where: · 4 unmodified lines
80	87	- new support lands as extensions
81	88	- source support does not require philosophy rewrites every few weeks
82	89
83		−That is the real reason the system is shaped around adapters, recipes, revisions, operations, and actor-scoped credentials.
	90	+That is the real reason the system is shaped around adapters, recipes, revisions, operations, and actor-scoped private state.

apps/docs/developer/mcp.mdx+2−2

@@ -9,7 +9,7 @@ MCP shows up in executor in two different ways. · 8 unmodified lines
9	9
10	10	This is the source side of MCP.
11	11
12		−When a user connects an MCP endpoint, executor treats it as an adapter with MCP-specific connection, discovery, and invocation behavior. The control plane persists the binding and credential state, while MCP-specific execution behavior lives behind the adapter and codemode layers.
	12	+When a user connects an MCP endpoint, executor treats it as an adapter with MCP-specific connection, discovery, invocation, and OAuth behavior. The control plane persists the binding and secret material state, while MCP-specific auth and execution behavior lives behind the adapter and codemode layers.
13	13
14	14	This is why MCP is modeled as its own adapter instead of being flattened into a generic HTTP execution model.
15	15
@@ -39,7 +39,7 @@ Keeping that distinction clear avoids collapsing very different responsibilities · 23 unmodified lines
39	39
40	40	Broadly:
41	41
42		−- control plane owns workspace state, bindings, credentials, and execution lifecycle
	42	+- control plane owns workspace state, bindings, secret material, and execution lifecycle
43	43	- `packages/hosts/mcp` owns the local MCP server surface
44	44	- `packages/platform/server` mounts the `/mcp` handler
45	45

apps/docs/developer/overview.mdx+4−4

@@ -7,7 +7,7 @@ Executor is built around one central idea: · 6 unmodified lines
7	7
8	8	- turn connected systems into a durable, discoverable tool catalog
9	9	- run TypeScript against that catalog inside a controlled runtime
10		−- keep connection state, auth, execution, and inspection in one local control plane
	10	+- keep connection state, secret material, execution, and inspection in one local control plane
11	11
12	12	## One-line architecture
13	13
@@ -16,7 +16,7 @@ Executor is a local control plane that: · 2 unmodified lines
16	16	1. stores connected integrations as sources
17	17	2. materializes those sources into callable tools
18	18	3. exposes those tools to the runtime, the UI, and MCP clients
19		−4. resolves auth and policy at invocation time
	19	+4. resolves secret material and policy at invocation time
20	20
21	21	## Major packages
22	22
@@ -48,7 +48,7 @@ The product core. · 25 unmodified lines
48	48	Responsibilities:
49	49
50	50	- source connection and lifecycle
51		−- auth and credential state
	51	+- secret material and private auth state
52	52	- persistence
53	53	- tool discovery and inspection
54	54	- execution management
@@ -76,7 +76,7 @@ That is why executor has strong internal concepts like: · 21 unmodified lines
76	76	- recipes and revisions
77	77	- materialized operations
78	78	- schema bundles
79		−- actor-scoped credentials
	79	+- actor-scoped secret material
80	80
81	81	Those are not UI concepts. They are the durable internal model that lets executor support multiple integration styles without rewriting the core every time.
82	82

bun.lock+1−0

@@ -683,6 +683,7 @@ · 682 unmodified lines
683	683	"name": "@executor/plugin-openapi-shared",
684	684	"version": "0.0.0",
685	685	"dependencies": {
	686	+ "@executor/platform-sdk": "workspace:*",
686	687	"@executor/source-core": "workspace:*",
687	688	"effect": "catalog:",
688	689	"yaml": "^2.8.1",

packages/platform/sdk/src/runtime/index.ts+0−1

@@ -99,7 +99,6 @@ export * from "./execution/live"; · 98 unmodified lines
99	99	export * from "./catalog/schema-type-signature";
100	100	export * from "./catalog/source/runtime";
101	101	export * from "./catalog/source/sync";
102		−export * from "./sources/source-credential-interactions";
103	102	export * from "./sources/source-store";
104	103	export * from "./executor-state-store";
105	104	export * from "./execution/scope/environment";

packages/platform/sdk/src/runtime/sources/source-credential-interactions.ts+0−34

@@ -1,34 +0,0 @@
1		−import {
2		− SecretRefSchema,
3		− type SecretRef,
4		−} from "#schema";
5		−import * as Schema from "effect/Schema";
6		−
7		−export const SourceCredentialSelectionContentSchema = Schema.Union(
8		− Schema.Struct({
9		− authKind: Schema.Literal("none"),
10		− }),
11		− Schema.Struct({
12		− authKind: Schema.Literal("bearer"),
13		− tokenRef: SecretRefSchema,
14		− }),
15		−);
16		−
17		−export type SourceCredentialSelectionContent =
18		− typeof SourceCredentialSelectionContentSchema.Type;
19		−
20		−export const decodeSourceCredentialSelectionContent = Schema.decodeUnknownSync(
21		− SourceCredentialSelectionContentSchema,
22		−);
23		−
24		−export const createSourceCredentialSelectionNoneContent =
25		− (): SourceCredentialSelectionContent => ({
26		− authKind: "none",
27		− });
28		−
29		−export const createSourceCredentialSelectionBearerContent = (
30		− tokenRef: SecretRef,
31		−): SourceCredentialSelectionContent => ({
32		− authKind: "bearer",
33		− tokenRef,
34		−});

packages/sources/core/src/shared.ts+0−20

@@ -1,4 +1,3 @@
1		−import * as Data from "effect/Data";
2	1	import * as Effect from "effect/Effect";
3	2
4	3	export const parseJsonValue = <T>(input: {
@@ -14,22 +13,3 @@ export const parseJsonValue = <T>(input: { · 9 unmodified lines
14	13	? new Error(`Invalid ${input.label}: ${cause.message}`)
15	14	: new Error(`Invalid ${input.label}: ${String(cause)}`),
16	15	});
17		−
18		−export class SourceCredentialRequiredError extends Data.TaggedError(
19		− "SourceCredentialRequiredError",
20		−)<{
21		− readonly slot: "runtime" \| "import";
22		− readonly message: string;
23		−}> {
24		− constructor(
25		− slot: "runtime" \| "import",
26		− message: string,
27		− ) {
28		− super({ slot, message });
29		− }
30		−}
31		−
32		−export const isSourceCredentialRequiredError = (
33		− error: unknown,
34		−): error is SourceCredentialRequiredError =>
35		− error instanceof SourceCredentialRequiredError;

plugins/google-discovery/sdk/index.ts+1−3

@@ -10,7 +10,6 @@ import { · 9 unmodified lines
10	10	import {
11	11	createCatalogImportMetadata,
12	12	createSourceCatalogSyncResult,
13		− SourceCredentialRequiredError,
14	13	} from "@executor/source-core";
15	14	import type { Source } from "@executor/platform-sdk/schema";
16	15	import type {
@@ -294,8 +293,7 @@ const fetchGoogleDiscoveryDocumentWithHeaders = (input: { · 277 unmodified lines
294	293	signal: AbortSignal.timeout(15_000),
295	294	});
296	295	if (response.status === 401 \|\| response.status === 403) {
297		− throw new SourceCredentialRequiredError(
298		− "import",
	296	+ throw new Error(
299	297	`Google Discovery fetch requires credentials (status ${response.status})`,
300	298	);
301	299	}

plugins/graphql/sdk/index.ts+1−3

@@ -9,7 +9,6 @@ import * as Schema from "effect/Schema"; · 8 unmodified lines
9	9	import {
10	10	createCatalogImportMetadata,
11	11	createSourceCatalogSyncResult,
12		− SourceCredentialRequiredError,
13	12	} from "@executor/source-core";
14	13	import type { Source } from "@executor/platform-sdk/schema";
15	14	import type {
@@ -200,8 +199,7 @@ const fetchGraphqlIntrospectionDocumentWithHeaders = (input: { · 184 unmodified lines
200	199
201	200	if (!response.ok) {
202	201	if (response.status === 401 \|\| response.status === 403) {
203		− throw new SourceCredentialRequiredError(
204		− "import",
	202	+ throw new Error(
205	203	`GraphQL introspection requires credentials (status ${response.status})`,
206	204	);
207	205	}

plugins/openapi/react/index.tsx+36−14

@@ -38,7 +38,7 @@ import { · 37 unmodified lines
38	38	type SourceToolExplorerSearch,
39	39	useSourcePluginNavigation,
40	40	useSourcePluginSearch,
41		−} from "@executor/react/source-plugins";
	41	+} from "@executor/react/plugins";
42	42	import {
43	43	openApiHttpApiExtension,
44	44	} from "@executor/plugin-openapi-http";
@@ -169,6 +169,31 @@ const inputFromConfig = ( · 124 unmodified lines
169	169	auth: config.auth,
170	170	});
171	171
	172	+const secretValue = (input: OpenApiConnectInput["auth"]): string =>
	173	+ input.kind === "bearer"
	174	+ ? JSON.stringify(input.tokenSecretRef)
	175	+ : "";
	176	+
	177	+const authFromSecretValue = (
	178	+ authKind: OpenApiConnectInput["auth"]["kind"],
	179	+ value: string,
	180	+): OpenApiConnectInput["auth"] => {
	181	+ if (authKind === "none") {
	182	+ return {
	183	+ kind: "none",
	184	+ };
	185	+ }
	186	+
	187	+ if (!value) {
	188	+ throw new Error("Select a secret for bearer auth.");
	189	+ }
	190	+
	191	+ return {
	192	+ kind: "bearer",
	193	+ tokenSecretRef: JSON.parse(value) as OpenApiConnectInput["auth"] & { tokenSecretRef: never }["tokenSecretRef"],
	194	+ };
	195	+};
	196	+
172	197	const useAvailableSecrets = (
173	198	openApiHttpClient: ReturnType<typeof getOpenApiHttpClient>,
174	199	) => {
@@ -205,9 +230,7 @@ function OpenApiSourceForm(props: { · 30 unmodified lines
205	230	props.initialValue.auth.kind,
206	231	);
207	232	const [tokenSecretRef, setTokenSecretRef] = useState(
208		− props.initialValue.auth.kind === "bearer"
209		− ? props.initialValue.auth.tokenSecretRef
210		− : "",
	233	+ secretValue(props.initialValue.auth),
211	234	);
212	235	const [error, setError] = useState<string \| null>(null);
213	236	const [preview, setPreview] = useState<OpenApiPreviewResponse \| null>(null);
@@ -309,19 +332,12 @@ function OpenApiSourceForm(props: { · 95 unmodified lines
309	332	}
310	333
311	334	try {
	335	+ const auth = authFromSecretValue(authKind, tokenSecretRef);
312	336	await submitMutation.mutateAsync({
313	337	name: trimmedName,
314	338	specUrl: trimmedSpecUrl,
315	339	baseUrl: trimmedBaseUrl \|\| null,
316		− auth:
317		− authKind === "bearer"
318		− ? {
319		− kind: "bearer",
320		− tokenSecretRef: tokenSecretRef.trim(),
321		− }
322		− : {
323		− kind: "none",
324		− },
	340	+ auth,
325	341	});
326	342	} catch (cause) {
327	343	setError(cause instanceof Error ? cause.message : "Failed saving source.");
@@ -398,7 +414,13 @@ function OpenApiSourceForm(props: { · 70 unmodified lines
398	414	>
399	415	<option value="">Select a secret</option>
400	416	{availableSecrets.map((secret) => (
401		− <option key={secret.id} value={secret.id}>
	417	+ <option
	418	+ key={`${secret.providerId}:${secret.id}`}
	419	+ value={JSON.stringify({
	420	+ providerId: secret.providerId,
	421	+ handle: secret.id,
	422	+ })}
	423	+ >
402	424	{secret.name \|\| secret.id}
403	425	</option>
404	426	))}

plugins/openapi/sdk/index.ts+13−21

@@ -13,6 +13,9 @@ import type { · 12 unmodified lines
13	13	ExecutorSourceConnector,
14	14	SourcePluginRuntime,
15	15	} from "@executor/platform-sdk/plugins";
	16	+import {
	17	+ SecretMaterialResolverService,
	18	+} from "@executor/platform-sdk/runtime";
16	19	import {
17	20	OpenApiConnectionAuthSchema,
18	21	deriveOpenApiNamespace,
@@ -64,12 +67,6 @@ export type OpenApiSourceStorage = { · 45 unmodified lines
64	67	}) => Effect.Effect<void, Error, never>;
65	68	};
66	69
67		−export type OpenApiSecrets = {
68		− resolve: (input: {
69		− ref: string;
70		− }) => Effect.Effect<string, Error, never>;
71		−};
72		−
73	70	export type OpenApiSdk = {
74	71	previewDocument: (
75	72	input: OpenApiPreviewRequest,
@@ -93,7 +90,6 @@ export type OpenApiSdk = { · 17 unmodified lines
93	90
94	91	export type OpenApiSdkPluginOptions = {
95	92	storage: OpenApiSourceStorage;
96		− secrets?: OpenApiSecrets;
97	93	};
98	94
99	95	const OpenApiExecutorAddInputSchema = Schema.Struct({
@@ -414,22 +410,18 @@ const decodeResponseBody = async (response: Response): Promise<unknown> => { · 314 unmodified lines
414	410	};
415	411
416	412	const resolveBearerToken = (
417		− options: OpenApiSdkPluginOptions,
418	413	stored: OpenApiStoredSourceData,
419		−): Effect.Effect<string \| null, Error, never> => {
420		− if (stored.auth.kind === "none") {
	414	+): Effect.Effect<string \| null, Error, any> => {
	415	+ const { auth } = stored;
	416	+ if (auth.kind === "none") {
421	417	return Effect.succeed(null);
422	418	}
423	419
424		− if (!options.secrets) {
425		− return Effect.fail(
426		− new Error("OpenAPI bearer auth is configured, but no secret resolver is available."),
427		− );
428		− }
429		−
430		− return options.secrets.resolve({
431		− ref: stored.auth.tokenSecretRef,
432		− }).pipe(Effect.map((token) => token.trim()));
	420	+ return Effect.flatMap(SecretMaterialResolverService, (resolveSecretMaterial) =>
	421	+ resolveSecretMaterial({
	422	+ ref: auth.tokenSecretRef,
	423	+ }).pipe(Effect.map((token) => token.trim()))
	424	+ );
433	425	};
434	426
435	427	const openApiDocumentHeaders = (input: {
@@ -550,7 +542,7 @@ const createOpenApiSourceRuntime = ( · 114 unmodified lines
550	542	});
551	543	}
552	544
553		− const bearerToken = yield* resolveBearerToken(options, stored);
	545	+ const bearerToken = yield* resolveBearerToken(stored);
554	546	const fetched = yield* fetchOpenApiDocument({
555	547	stored,
556	548	bearerToken,
@@ -686,7 +678,7 @@ const createOpenApiSourceRuntime = ( · 129 unmodified lines
686	678	}
687	679	}
688	680
689		− const bearerToken = yield* resolveBearerToken(options, stored);
	681	+ const bearerToken = yield* resolveBearerToken(stored);
690	682	if (bearerToken && bearerToken.length > 0) {
691	683	headers.authorization = `Bearer ${bearerToken}`;
692	684	}

plugins/openapi/shared/index.ts+3−2

@@ -3,8 +3,9 @@ import * as Schema from "effect/Schema"; · 2 unmodified lines
3	3	import YAML from "yaml";
4	4
5	5	import {
	6	+ SecretRefSchema,
6	7	StringMapSchema,
7		−} from "@executor/source-core";
	8	+} from "@executor/platform-sdk/schema";
8	9
9	10	export const OpenApiConnectionAuthSchema = Schema.Union(
10	11	Schema.Struct({
@@ -12,7 +13,7 @@ export const OpenApiConnectionAuthSchema = Schema.Union( · 1 unmodified line
12	13	}),
13	14	Schema.Struct({
14	15	kind: Schema.Literal("bearer"),
15		− tokenSecretRef: Schema.String,
	16	+ tokenSecretRef: SecretRefSchema,
16	17	}),
17	18	);
18	19

plugins/openapi/shared/package.json+1−0

@@ -12,6 +12,7 @@ · 11 unmodified lines
12	12	"typecheck": "bunx tsc --noEmit -p tsconfig.json"
13	13	},
14	14	"dependencies": {
	15	+ "@executor/platform-sdk": "workspace:*",
15	16	"@executor/source-core": "workspace:*",
16	17	"effect": "catalog:",
17	18	"yaml": "^2.8.1"