Trace the OAuth lifecycle: token requests, connects, and health checks (#1485)

The OAuth domain layer had no spans. The token POST also bypasses Effect's HttpClient (oauth4webapi drives the raw fetch), so not even an http.client span existed underneath — AS latency and refusal rates were unmeasurable, and health verdicts were only obtainable by querying the database. - executor.oauth.token_request on all three grants: grant type, token hostname, client auth method, and the AS's RFC 6749 error code on failure. Hostname only, never the full URL. - executor.oauth.complete around the code redemption: integration, client slug, whether the token host was rebound to a regional endpoint, and the connection's freshness facts at birth (refresh token present, expiry advertised, scope shortfall counts). - executor.connection.health.check / executor.connection.validate with the verdict, its source (cache, probe, credential-only, no-capability), and http status. mcp.plugin.check_health gets the same outcome stamp — its failures fold onto the success channel, so the span previously ended green on an expired credential. - The AS error code is now a typed oauthErrorCode field on CredentialResolutionError and a structured detail on the sandbox-facing failure, instead of surviving only inside message text.

Rhys Sullivan committed 3 months ago

Commit f78b8ca1 parent b7c23337 files · +217 −25Browse files

Unified · Split

7 files

e2e/scenarios/oauth-refresh-rejected.test.ts +5 −0
packages/core/execution/src/tool-invoker.ts +8 −0
packages/core/sdk/src/errors.ts +7 −0
packages/core/sdk/src/executor.ts +72 −16
packages/core/sdk/src/oauth-helpers.ts +64 −3
packages/core/sdk/src/oauth-service.ts +44 −5
packages/plugins/mcp/src/sdk/plugin.ts +17 −1

e2e/scenarios/oauth-refresh-rejected.test.ts+5−0

     readonly details?: {
       readonly category?: string;
       readonly recovery?: Record<string, string>;
+      readonly upstream?: { readonly details?: { readonly oauthErrorCode?: string } };
     };
   };
 };
             failure.error?.message ?? "",
             "the message carries the AS's RFC 6749 error code",
           ).toContain("invalid_request");
+          expect(
+            failure.error?.details?.upstream?.details?.oauthErrorCode,
+            "the AS's code is ALSO a structured field, so agents and telemetry never parse the message for it",
+          ).toBe("invalid_request");
           expect(
             failure.error?.message ?? "",
             "the message carries the AS's error description",

packages/core/execution/src/tool-invoker.ts+8−0

   readonly label: string;
   readonly message: string;
   readonly reauthRequired?: boolean;
+  readonly oauthErrorCode?: string;
 }) =>
   authToolFailure({
     code: input.reauthRequired === true ? "oauth_reauth_required" : "oauth_refresh_failed",
       kind: "oauth",
       label: input.label,
     },
+    // The AS's own RFC 6749 §5.2 verdict, when there was one — structured so
+    // the agent (and anyone reading the failure) can distinguish a dead grant
+    // from a misconfigured app without parsing the message.
+    ...(input.oauthErrorCode !== undefined
+      ? { upstream: { details: { oauthErrorCode: input.oauthErrorCode } } }
+      : {}),
   });
 const bindingToolFailure = (value: unknown): ToolError | null => {
             label: `${err.integration}.${err.owner}.${err.name}`,
             message: err.message,
             reauthRequired: err.reauthRequired,
+            oauthErrorCode: err.oauthErrorCode,
           }),
         ),
       ),

packages/core/sdk/src/errors.ts+7−0

@@ -181,6 +181,13 @@ export class CredentialResolutionError extends Schema.TaggedErrorClass<Credentia · 180 unmodified lines
181	/** True when the stored grant is permanently invalid and the user must	181	/** True when the stored grant is permanently invalid and the user must
182	* sign in again (RFC 6749 §5.2 invalid_grant and friends). */	182	* sign in again (RFC 6749 §5.2 invalid_grant and friends). */
183	reauthRequired: Schema.optional(Schema.Boolean),	183	reauthRequired: Schema.optional(Schema.Boolean),
		184	/** The authorization server's RFC 6749 §5.2 error code (`invalid_grant`,
		185	* `invalid_client`, …), when the failure came from a token-endpoint
		186	* refusal. A typed field rather than message text so telemetry and
		187	* classification read it structurally — the misclassification where
		188	* `invalid_client` (rotated app secret, fleet-wide) surfaced as a vague
		189	* "degraded" was only findable by grepping persisted message strings. */
		190	oauthErrorCode: Schema.optional(Schema.String),
184	},	191	},
185	) {}	192	) {}
186		193

packages/core/sdk/src/executor.ts+72−16

                         // oxlint-disable-next-line executor/no-unknown-error-message -- boundary: OAuth2Error carries a typed `message`
                         message: `OAuth token refresh was rejected (${cause.error}): ${cause.message}`,
                         reauthRequired: cause.error === "invalid_grant",
+                        oauthErrorCode: cause.error,
                       });
                     }
                     return new StorageError({
         // The refresh path was previously invisible to telemetry: no span, no
         // log, no metric. When a customer reported "my OAuth just died", there
         // was no way to answer "did a refresh even fire, and did it work?"
-        // without a repro. Stamp the outcome and the failure KIND — enumerable
+        // without a repro. Stamp the outcome, the failure KIND, and the AS's
-        // identifiers only, never user content or token material.
+        // own RFC 6749 §5.2 code (from the typed `oauthErrorCode` field, NOT
-        //
+        // the message — the message embeds the token endpoint's response URL
-        // The AS's own RFC 6749 §5.2 code is not stamped here: it survives only
+        // and a body preview). Enumerable identifiers only, never user content
-        // inside the error's message, and that message embeds the token
+        // or token material. The code is the dimension that separates
-        // endpoint's response URL and a body preview, so it can't go on a span
+        // invalid_client (a rotated app secret — fleet-wide, page someone)
-        // attribute. Making that code a queryable dimension needs it lifted to
+        // from server_error (transient, the next invoke retries).
-        // a typed field on CredentialResolutionError first — worth doing, since
-        // invalid_client (a rotated secret, fleet-wide) currently looks exactly
-        // like a transient server_error from a trace.
         Effect.tap(() => Effect.annotateCurrentSpan({ "executor.oauth.refresh.outcome": "ok" })),
         Effect.tapError((error: StorageFailure | CredentialResolutionError) =>
           Effect.annotateCurrentSpan({
             // the refresh token itself is dead) or a transient failure the next
             // invoke can retry. The split is the actionable half of the signal.
             ...(Predicate.isTagged(error, "CredentialResolutionError")
-              ? { "executor.oauth.refresh.reauth_required": error.reauthRequired === true }
+              ? {
+                  "executor.oauth.refresh.reauth_required": error.reauthRequired === true,
+                  ...(error.oauthErrorCode !== undefined
+                    ? { "executor.oauth.error_code": error.oauthErrorCode }
+                    : {}),
+                }
               : {}),
           }),
         ),
         return out;
       });
+    /** Stamp the verdict + which path produced it onto the enclosing health
+     *  span. Every health outcome is HTTP 200, so WITHOUT these attributes the
+     *  request envelope cannot separate healthy from expired — the "what
+     *  fraction of connections are dead right now" question was previously
+     *  only answerable by querying the database. `status` and `httpStatus`
+     *  are enumerable; `detail`/`identity` (upstream free text / an email)
+     *  never go on a span. */
+    const annotateHealthVerdict = (
+      source: "cache" | "no_capability" | "credential_only" | "probe",
+      result: HealthCheckResult,
+    ): Effect.Effect<void> =>
+      Effect.annotateCurrentSpan({
+        "executor.health.status": result.status,
+        "executor.health.source": source,
+        ...(result.httpStatus !== undefined
+          ? { "executor.health.http_status": result.httpStatus }
+          : {}),
+      });
     const connectionCheckHealth = (
       ref: ConnectionRef,
       options?: {
         }
         if (options?.ifStaleMs !== undefined) {
           const cached = Option.getOrNull(decodeLastHealth(connectionRow.last_health));
-          if (cached && Date.now() - cached.checkedAt < options.ifStaleMs) return cached;
+          if (cached && Date.now() - cached.checkedAt < options.ifStaleMs) {
+            yield* annotateHealthVerdict("cache", cached);
+            return cached;
+          }
         }
         const integrationRow = yield* findIntegrationRow(ref.integration);
         if (!integrationRow) {
         }
         const runtime = runtimes.get(integrationRow.plugin_id);
         const check = runtime?.plugin.checkHealth;
-        if (!runtime || !check) return unknownHealth();
+        if (!runtime || !check) {
+          const result = unknownHealth();
+          yield* annotateHealthVerdict("no_capability", result);
+          return result;
+        }
         const spec = describeHealthCheckForRow(integrationRow) ?? undefined;
         if (spec === undefined && connectionRow.oauth_client != null) {
+          // No probe operation is declared, so "healthy" here means only "the
+          // credential resolved (refreshing if due)" — a refresh failure is
+          // the one real signal this path can produce, and it must not hide
+          // inside a green span.
           const result = yield* oauthCredentialHealthWithoutProbe(connectionRow);
+          yield* annotateHealthVerdict("credential_only", result);
           yield* persistHealthResult(ref, result);
           return result;
         }
             `Health check for connection "${ref.name}" failed.`,
           );
         }).pipe(Effect.catchTag("CredentialResolutionError", healthFromCredentialResolutionError));
+        yield* annotateHealthVerdict("probe", result);
         // Persist the verdict on the connection row so the accounts list shows
         // alive/expired at a glance. Best-effort: a write failure must not turn
         // a successful probe into an error.
         yield* persistHealthResult(ref, result);
         return result;
-      });
+      }).pipe(
+        Effect.withSpan("executor.connection.health.check", {
+          attributes: {
+            "executor.integration": String(ref.integration),
+            "executor.connection": String(ref.name),
+          },
+        }),
+      );
     const connectionValidate = (
       input: ValidateConnectionInput,
         // from the integration row. Nothing persists here: validate is the
         // key-first flow's dry run.
         const spec = input.spec ?? describeHealthCheckForRow(integrationRow) ?? undefined;
-        return yield* foldPluginFailure(
+        const result = yield* foldPluginFailure(
           check({ ctx: runtime.ctx, integration: record, credential, spec }),
           `Validating credential for "${input.integration}" failed.`,
         );
-      });
+        // Nothing persists here BY DESIGN, which makes this span the only
+        // possible record of "what fraction of pasted credentials are rejected
+        // at the door" — a signal the DB can never carry.
+        yield* Effect.annotateCurrentSpan({
+          "executor.health.status": result.status,
+          ...(result.httpStatus !== undefined
+            ? { "executor.health.http_status": result.httpStatus }
+            : {}),
+        });
+        return result;
+      }).pipe(
+        Effect.withSpan("executor.connection.validate", {
+          attributes: { "executor.integration": String(input.integration) },
+        }),
+      );
     // Clear the sync stamp so the next tools read re-produces this connection's
     // catalog. The deferred variant of `connectionsRefresh` for signals that

packages/core/sdk/src/oauth-helpers.ts+64−3

@@ -367,6 +367,43 @@ const toOAuth2ErrorWithHttpSummary = (cause: unknown): Effect.Effect<OAuth2Error · 366 unmodified lines
367	const failOAuth2WithHttpSummary = (cause: unknown): Effect.Effect<never, OAuth2Error> =>	367	const failOAuth2WithHttpSummary = (cause: unknown): Effect.Effect<never, OAuth2Error> =>
368	toOAuth2ErrorWithHttpSummary(cause).pipe(Effect.flatMap((error) => Effect.fail(error)));	368	toOAuth2ErrorWithHttpSummary(cause).pipe(Effect.flatMap((error) => Effect.fail(error)));
369		369
		370	/** Trace one token-endpoint round trip. This is the ONLY place a token request
		371	* can be observed: oauth4webapi drives the raw global `fetch`, not Effect's
		372	* HttpClient, so no `http.client` span exists underneath — without this span
		373	* the AS's latency and refusal rate are invisible.
		374	*
		375	* Attribute discipline: hostname (never the full URL — some providers carry
		376	* tenant ids in the path), the grant literal, the auth method, and on failure
		377	* the AS's RFC 6749 §5.2 code. Never the OAuth2Error message (it embeds the
		378	* response URL and a body preview), never token or code material. */
		379	const withTokenRequestSpan =
		380	(input: {
		381	readonly grantType: "authorization_code" \| "client_credentials" \| "refresh_token";
		382	readonly tokenUrl: string;
		383	readonly clientAuth: ClientAuthMethod \| undefined;
		384	readonly hasResource: boolean;
		385	}) =>
		386	<A>(effect: Effect.Effect<A, OAuth2Error>): Effect.Effect<A, OAuth2Error> =>
		387	effect.pipe(
		388	Effect.tapError((error) =>
		389	Effect.annotateCurrentSpan({
		390	...(error.error !== undefined ? { "executor.oauth.error_code": error.error } : {}),
		391	}),
		392	),
		393	Effect.withSpan("executor.oauth.token_request", {
		394	attributes: {
		395	"executor.oauth.grant_type": input.grantType,
		396	"executor.oauth.token_host": hostnameForTelemetry(input.tokenUrl),
		397	"executor.oauth.client_auth": input.clientAuth ?? DEFAULT_CLIENT_AUTH_METHOD,
		398	"executor.oauth.has_resource": input.hasResource,
		399	},
		400	}),
		401	);
		402
		403	/** The hostname alone — a malformed URL yields "invalid" rather than leaking
		404	* whatever string failed to parse. */
		405	const hostnameForTelemetry = (url: string): string => URL.parse(url)?.hostname ?? "invalid";
		406
370	// ---------------------------------------------------------------------------	407	// ---------------------------------------------------------------------------
371	// oauth4webapi adapter helpers	408	// oauth4webapi adapter helpers
372	// ---------------------------------------------------------------------------	409	// ---------------------------------------------------------------------------
@@ -614,7 +651,15 @@ export const exchangeAuthorizationCode = ( · 241 unmodified lines
614	return await processTokenEndpointResponse(as, client, response);	651	return await processTokenEndpointResponse(as, client, response);
615	},	652	},
616	catch: (cause) => cause,	653	catch: (cause) => cause,
617	}).pipe(Effect.catch(failOAuth2WithHttpSummary));	654	}).pipe(
		655	Effect.catch(failOAuth2WithHttpSummary),
		656	withTokenRequestSpan({
		657	grantType: "authorization_code",
		658	tokenUrl: input.tokenUrl,
		659	clientAuth: input.clientAuth,
		660	hasResource: input.resource !== undefined,
		661	}),
		662	);
618		663
619	// ---------------------------------------------------------------------------	664	// ---------------------------------------------------------------------------
620	// Exchange client credentials → tokens (RFC 6749 §4.4)	665	// Exchange client credentials → tokens (RFC 6749 §4.4)
@@ -669,7 +714,15 @@ export const exchangeClientCredentials = ( · 48 unmodified lines
669	return tokenResponseFrom(result);	714	return tokenResponseFrom(result);
670	},	715	},
671	catch: (cause) => cause,	716	catch: (cause) => cause,
672	}).pipe(Effect.catch(failOAuth2WithHttpSummary));	717	}).pipe(
		718	Effect.catch(failOAuth2WithHttpSummary),
		719	withTokenRequestSpan({
		720	grantType: "client_credentials",
		721	tokenUrl: input.tokenUrl,
		722	clientAuth: input.clientAuth,
		723	hasResource: input.resource !== undefined,
		724	}),
		725	);
673		726
674	// ---------------------------------------------------------------------------	727	// ---------------------------------------------------------------------------
675	// Refresh access token	728	// Refresh access token
@@ -740,7 +793,15 @@ export const refreshAccessToken = ( · 64 unmodified lines
740	return tokenResponseFrom(result);	793	return tokenResponseFrom(result);
741	},	794	},
742	catch: (cause) => cause,	795	catch: (cause) => cause,
743	}).pipe(Effect.catch(failOAuth2WithHttpSummary));	796	}).pipe(
		797	Effect.catch(failOAuth2WithHttpSummary),
		798	withTokenRequestSpan({
		799	grantType: "refresh_token",
		800	tokenUrl: input.tokenUrl,
		801	clientAuth: input.clientAuth,
		802	hasResource: input.resource !== undefined,
		803	}),
		804	);
744		805
745	// ---------------------------------------------------------------------------	806	// ---------------------------------------------------------------------------
746	// Refresh-needed predicate	807	// Refresh-needed predicate

packages/core/sdk/src/oauth-service.ts+44−5

@@ -1283,6 +1283,15 @@ export const makeOAuthService = (deps: OAuthServiceDeps): OAuthService => { · 1282 unmodified lines
1283	clientOwnerFromPayload(sessionRow.payload) ?? (String(sessionRow.owner) as Owner),	1283	clientOwnerFromPayload(sessionRow.payload) ?? (String(sessionRow.owner) as Owner),
1284	};	1284	};
1285		1285
		1286	// Annotate as soon as the session resolves the flow's identity, so even
		1287	// a completion that fails at the exchange still says WHOSE connect died.
		1288	yield* Effect.annotateCurrentSpan({
		1289	"executor.integration": String(session.integration),
		1290	"executor.connection": String(session.name),
		1291	"executor.template": String(session.template),
		1292	"executor.oauth.client": String(session.clientSlug),
		1293	});
		1294
1286	// Expired sessions are not redeemable — drop + treat as not found.	1295	// Expired sessions are not redeemable — drop + treat as not found.
1287	if (Number.isFinite(session.expiresAt) && session.expiresAt <= Date.now()) {	1296	if (Number.isFinite(session.expiresAt) && session.expiresAt <= Date.now()) {
1288	yield* deleteSession(input.state);	1297	yield* deleteSession(input.state);
@@ -1368,9 +1377,24 @@ export const makeOAuthService = (deps: OAuthServiceDeps): OAuthService => { · 79 unmodified lines
1368	),	1377	),
1369	);	1378	);
1370		1379
		1380	// Everything a "why did this connect fail / where did it go" question
		1381	// needs, none of it secret: slugs, owner scope, and whether the token
		1382	// host was rebound to a regional endpoint (the Datadog multi-site path —
		1383	// a bug there previously shipped and was only diagnosable by hand).
		1384	// Deliberately absent: the code, the PKCE verifier, the token, the
		1385	// callback domain (can embed an org's private site), and identityLabel
		1386	// (resolves to an email).
		1387	yield* Effect.annotateCurrentSpan({
		1388	"executor.oauth.token_host_rebound": tokenUrl !== client.tokenUrl,
		1389	});
		1390
1371	yield* deleteSession(input.state);	1391	yield* deleteSession(input.state);
1372	return connection;	1392	return connection;
1373	});	1393	}).pipe(
		1394	Effect.withSpan("executor.oauth.complete", {
		1395	attributes: { "executor.oauth.grant": "authorization_code" },
		1396	}),
		1397	);
1374		1398
1375	// -----------------------------------------------------------------------	1399	// -----------------------------------------------------------------------
1376	// Mint the connection from a freshly exchanged token: store the access	1400	// Mint the connection from a freshly exchanged token: store the access
@@ -1416,6 +1440,24 @@ export const makeOAuthService = (deps: OAuthServiceDeps): OAuthService => { · 39 unmodified lines
1416	}	1440	}
1417		1441
1418	const oauthScope = recordedOAuthScope(token, requestedScopes);	1442	const oauthScope = recordedOAuthScope(token, requestedScopes);
		1443	const missingScopes =
		1444	client.grant === "authorization_code"
		1445	? missingGrantedOAuthScopes(requestedScopes, oauthScope)
		1446	: [];
		1447	// The freshness facts of this connection AT BIRTH, on the enclosing
		1448	// span (executor.oauth.complete, or the reconnect path's request
		1449	// envelope). Every "why did this connection later go stale" question
		1450	// starts here: a partial grant fails later as oauth_scope_insufficient
		1451	// in an unrelated trace; no refresh token means the first expiry is
		1452	// terminal; no advertised expiry means only the reactive 401 path can
		1453	// ever refresh it. Counts and booleans only — scope VALUES can encode
		1454	// customer resource names on some providers.
		1455	yield* Effect.annotateCurrentSpan({
		1456	"executor.oauth.scope_requested_count": requestedScopes.length,
		1457	"executor.oauth.scope_missing_count": missingScopes.length,
		1458	"executor.oauth.has_refresh_token": token.refresh_token !== undefined,
		1459	"executor.oauth.has_advertised_expiry": typeof token.expires_in === "number",
		1460	});
1419	return yield* deps.mintOAuthConnection({	1461	return yield* deps.mintOAuthConnection({
1420	owner: target.owner,	1462	owner: target.owner,
1421	name: target.name,	1463	name: target.name,
@@ -1436,10 +1478,7 @@ export const makeOAuthService = (deps: OAuthServiceDeps): OAuthService => { · 14 unmodified lines
1436	// non-resource scope from the token `scope` string, so preserve it when	1478	// non-resource scope from the token `scope` string, so preserve it when
1437	// the refresh token proves it was granted.	1479	// the refresh token proves it was granted.
1438	oauthScope,	1480	oauthScope,
1439	missingOAuthScopes:	1481	missingOAuthScopes: missingScopes,
1440	client.grant === "authorization_code"
1441	? missingGrantedOAuthScopes(requestedScopes, oauthScope)
1442	: [],
1443	oauthTokenUrl,	1482	oauthTokenUrl,
1444	});	1483	});
1445	});	1484	});

packages/plugins/mcp/src/sdk/plugin.ts+17−1

@@ -1555,7 +1555,23 @@ export const mcpPlugin = definePlugin((options?: McpPluginOptions) => { · 1554 unmodified lines
1555	detail: error.message,	1555	detail: error.message,
1556	} satisfies HealthCheckResult),	1556	} satisfies HealthCheckResult),
1557	),	1557	),
1558	Effect.withSpan("mcp.plugin.check_health"),	1558	// Every failure above folds onto the SUCCESS channel, so without this
		1559	// the span ends green on an expired credential — a 401 auth wall and a
		1560	// healthy dial were indistinguishable in traces.
		1561	Effect.tap((result) =>
		1562	Effect.annotateCurrentSpan({
		1563	"mcp.health.status": result.status,
		1564	...("httpStatus" in result && result.httpStatus !== undefined
		1565	? { "mcp.health.http_status": result.httpStatus }
		1566	: {}),
		1567	}),
		1568	),
		1569	Effect.withSpan("mcp.plugin.check_health", {
		1570	attributes: {
		1571	"executor.integration": String(credential.integration),
		1572	"mcp.connection.name": String(credential.connection),
		1573	},
		1574	}),
1559	),	1575	),
1560		1576
1561	describeAuthMethods: describeMcpAuthMethods,	1577	describeAuthMethods: describeMcpAuthMethods,