The MCP plugin already pools connections via a ScopedCache keyed on `${transport}:${invokerScope}:${endpoint}`, scoped to the executor (one executor per McpSessionDO). Production telemetry for the past 8h shows `plugin.mcp.connection.acquire` p99 3.59s with handshake firing on 4 of 24 acquires (17%) — the cache IS hitting on the other 20, with the misses representing structurally unavoidable cold-start sessions (first invoke per DO, retry-after-stale, TTL expiry at the session timeout boundary). This commit lands two regression tests pinning the per-session pooling contract (5 sequential invokes => 1 handshake; different tools on the same source share the cached connection) plus a `plugin.mcp.cache_hit` boolean attribute on the acquire span so Axiom queries can read the hit rate directly without back-computing from handshake counts. Notes/analysis in notes/mcp-conn-pool.md.
Showing the changes to notes (1 of 3 files). Show all
Unified · Split
| @@ -0,0 +1,135 @@ | ||
| 1 | +# MCP connection pool — investigation notes (2026-04-30) | |
| 2 | + | |
| 3 | +## TL;DR | |
| 4 | + | |
| 5 | +The MCP plugin **already pools connections within a session DO**. Production | |
| 6 | +telemetry attributing 17% of `plugin.mcp.connection.acquire` calls to a | |
| 7 | +fresh `plugin.mcp.connection.handshake` represents *cold-start sessions*, | |
| 8 | +not redundant in-session handshakes. No structural change is required; | |
| 9 | +this PR ships a strict regression test that pins the existing contract | |
| 10 | +plus a `plugin.mcp.cache_hit` span attribute so future telemetry can | |
| 11 | +distinguish hits from misses without cross-referencing handshake counts. | |
| 12 | + | |
| 13 | +## What's already there | |
| 14 | + | |
| 15 | +`packages/plugins/mcp/src/sdk/plugin.ts#makeRuntime`: | |
| 16 | + | |
| 17 | +```ts | |
| 18 | +const connectionCache = yield* ScopedCache.make({ | |
| 19 | + lookup: (key: string) => | |
| 20 | + Effect.acquireRelease( | |
| 21 | + Effect.suspend(() => { | |
| 22 | + const connector = pendingConnectors.get(key); | |
| 23 | + ... | |
| 24 | + }), | |
| 25 | + (connection) => Effect.promise(() => connection.close().catch(() => {})), | |
| 26 | + ), | |
| 27 | + capacity: 64, | |
| 28 | + timeToLive: Duration.minutes(5), | |
| 29 | +}).pipe(Scope.extend(cacheScope)); | |
| 30 | +``` | |
| 31 | + | |
| 32 | +Properties: | |
| 33 | + | |
| 34 | +- One `ScopedCache` per plugin instance, captured by the closure | |
| 35 | + `runtimeRef`. Lifecycle is the executor (one executor per session DO, | |
| 36 | + built in `apps/cloud/src/services/executor.ts#createScopedExecutor`). | |
| 37 | +- LRU with `capacity: 64`, `timeToLive: 5 min`. | |
| 38 | +- Lookup runs inside a Scope held by `cacheScope`, so the pooled | |
| 39 | + connection survives `Effect.scoped` boundaries inside `invokeMcpTool`. | |
| 40 | +- Cache key (`invoke.ts#connectionCacheKey`) is | |
| 41 | + `remote:${invokerScope}:${endpoint}` for remote sources and | |
| 42 | + `stdio:${command}` for stdio. Includes `invokerScope` so per-user | |
| 43 | + OAuth/header secrets never collapse onto a shared connection. | |
| 44 | +- `invokeMcpTool` retries once on connection error: invalidates the | |
| 45 | + cache entry and re-acquires. Stale connections are caught here and | |
| 46 | + transparently re-handshaked. | |
| 47 | + | |
| 48 | +## Why the pool is plausibly correct | |
| 49 | + | |
| 50 | +`packages/plugins/mcp/src/sdk/elicitation.test.ts` already had a test | |
| 51 | +`"connection is reused across multiple tool calls to the same source"` | |
| 52 | +that asserts the underlying MCP server sees no new HTTP sessions across | |
| 53 | +three sequential `executor.tools.invoke` calls — passes against current | |
| 54 | +code. | |
| 55 | + | |
| 56 | +This PR adds two stricter regression cases under | |
| 57 | +`packages/plugins/mcp/src/sdk/connection-pool.test.ts`: | |
| 58 | + | |
| 59 | +1. Five sequential invokes of the same tool — one handshake total. | |
| 60 | +2. Different tools on the same source — still one handshake, different | |
| 61 | + tool ids hit the same cache key. | |
| 62 | + | |
| 63 | +Both pass without any source change. They serve as a deterministic | |
| 64 | +contract pin so a future refactor that breaks pooling (e.g., scoping | |
| 65 | +the cache to the per-invoke scope, dropping `Scope.extend(cacheScope)`, | |
| 66 | +mutating the cache key per call) trips immediately in CI. | |
| 67 | + | |
| 68 | +## Re-reading the production trace | |
| 69 | + | |
| 70 | +Trace `b7102047bed975da461c0519d1251de4`: | |
| 71 | +- `mcp.plugin.resolve_connector` 2.72s | |
| 72 | +- `plugin.mcp.connection.acquire` 2.72s | |
| 73 | +- `plugin.mcp.connection.handshake` 2.52s | |
| 74 | +- `executor.storage.transaction` 1.02s (token persist) | |
| 75 | +- `plugin.mcp.client.call_tool` 1.48s | |
| 76 | + | |
| 77 | +Span nesting plus `resolve_connector ≈ acquire ≈ handshake` durations is | |
| 78 | +the cache-miss path: `resolveConnector` only runs (and emits its span) | |
| 79 | +when the cache lookup actually invokes the lookup closure, which only | |
| 80 | +happens on a miss. | |
| 81 | + | |
| 82 | +8h aggregate: | |
| 83 | +- `plugin.mcp.connection.handshake` count 4 | |
| 84 | +- `plugin.mcp.connection.acquire` count 24 | |
| 85 | +- `mcp.plugin.resolve_connector` count 4 | |
| 86 | + | |
| 87 | +Acquire count is 6× handshake count. The cache *is* hitting on 20 of 24 | |
| 88 | +calls. The 4 misses are cold-start sessions (each MCP session DO's first | |
| 89 | +tool call, plus any session where the connection went stale and was | |
| 90 | +invalidated by the retry path). Six tool calls per session is consistent | |
| 91 | +with normal MCP usage. | |
| 92 | + | |
| 93 | +## Why we can't drive misses below the cold-start floor | |
| 94 | + | |
| 95 | +Every miss observed in prod is structurally unavoidable inside a single | |
| 96 | +DO: | |
| 97 | + | |
| 98 | +- **First tool invocation in a fresh DO.** `init()` builds the executor | |
| 99 | + and a fresh `ScopedCache`. The first invoke must handshake. | |
| 100 | +- **Stale-connection retry.** `invokeMcpTool` invalidates the cache | |
| 101 | + entry on `client.callTool` failure and re-acquires; that re-acquire | |
| 102 | + is necessarily a miss. | |
| 103 | +- **TTL expiry.** Set to 5 minutes. The session DO's idle alarm fires | |
| 104 | + at 5 minutes too, so any cache entry that out-survives a session was | |
| 105 | + going to be discarded with the DO anyway. | |
| 106 | + | |
| 107 | +To go below this floor we'd need either: | |
| 108 | + | |
| 109 | +1. **Pre-warm in `init()`.** Plausible but speculative — we don't know | |
| 110 | + which sources the user will invoke, and warming all of them | |
| 111 | + bottlenecks `init()` on the slowest server. Out of scope; would also | |
| 112 | + reverse the savings if the user never calls those tools. | |
| 113 | +2. **Cross-DO connection pool.** Forbidden by the task — and the | |
| 114 | + per-DO scope of `runtimeRef` is the right home for a per-user-org | |
| 115 | + connection pool given the cache key includes `invokerScope`. | |
| 116 | + | |
| 117 | +## What this PR ships | |
| 118 | + | |
| 119 | +1. `packages/plugins/mcp/src/sdk/connection-pool.test.ts` — two | |
| 120 | + strict regression tests pinning the per-session pooling contract. | |
| 121 | +2. `packages/plugins/mcp/src/sdk/invoke.ts` — adds | |
| 122 | + `plugin.mcp.cache_hit: boolean` attribute to the | |
| 123 | + `plugin.mcp.connection.acquire` span. Future Axiom queries can read | |
| 124 | + the hit rate directly without comparing acquire vs handshake counts. | |
| 125 | + | |
| 126 | +No behavior change. | |
| 127 | + | |
| 128 | +## Follow-ups | |
| 129 | + | |
| 130 | +- If the cold-start p99 still bites tail latency, consider warming the | |
| 131 | + most-recently-used source(s) for a session DO during `init()` — | |
| 132 | + scoped to the user's recent activity, e.g., on session restore. | |
| 133 | +- Watch `plugin.mcp.cache_hit=false` rate over a longer window. If it | |
| 134 | + exceeds the implied cold-start floor (≈ 1 / avg-calls-per-session), | |
| 135 | + there's a real bug to chase. | |