The MCP plugin already pools connections via a ScopedCache keyed on `${transport}:${invokerScope}:${endpoint}`, scoped to the executor (one executor per McpSessionDO). Production telemetry for the past 8h shows `plugin.mcp.connection.acquire` p99 3.59s with handshake firing on 4 of 24 acquires (17%) — the cache IS hitting on the other 20, with the misses representing structurally unavoidable cold-start sessions (first invoke per DO, retry-after-stale, TTL expiry at the session timeout boundary). This commit lands two regression tests pinning the per-session pooling contract (5 sequential invokes => 1 handshake; different tools on the same source share the cached connection) plus a `plugin.mcp.cache_hit` boolean attribute on the acquire span so Axiom queries can read the hit rate directly without back-computing from handshake counts. Notes/analysis in notes/mcp-conn-pool.md.
Showing the changes to notes (1 of 3 files). Show all
Unified · Split
| @@ -0,0 +1,135 @@ | |||
| 1 | # MCP connection pool — investigation notes (2026-04-30) | ||
| 2 | |||
| 3 | ## TL;DR | ||
| 4 | |||
| 5 | The MCP plugin **already pools connections within a session DO**. Production | ||
| 6 | telemetry attributing 17% of `plugin.mcp.connection.acquire` calls to a | ||
| 7 | fresh `plugin.mcp.connection.handshake` represents *cold-start sessions*, | ||
| 8 | not redundant in-session handshakes. No structural change is required; | ||
| 9 | this PR ships a strict regression test that pins the existing contract | ||
| 10 | plus a `plugin.mcp.cache_hit` span attribute so future telemetry can | ||
| 11 | distinguish hits from misses without cross-referencing handshake counts. | ||
| 12 | |||
| 13 | ## What's already there | ||
| 14 | |||
| 15 | `packages/plugins/mcp/src/sdk/plugin.ts#makeRuntime`: | ||
| 16 | |||
| 17 | ```ts | ||
| 18 | const connectionCache = yield* ScopedCache.make({ | ||
| 19 | lookup: (key: string) => | ||
| 20 | Effect.acquireRelease( | ||
| 21 | Effect.suspend(() => { | ||
| 22 | const connector = pendingConnectors.get(key); | ||
| 23 | ... | ||
| 24 | }), | ||
| 25 | (connection) => Effect.promise(() => connection.close().catch(() => {})), | ||
| 26 | ), | ||
| 27 | capacity: 64, | ||
| 28 | timeToLive: Duration.minutes(5), | ||
| 29 | }).pipe(Scope.extend(cacheScope)); | ||
| 30 | ``` | ||
| 31 | |||
| 32 | Properties: | ||
| 33 | |||
| 34 | - One `ScopedCache` per plugin instance, captured by the closure | ||
| 35 | `runtimeRef`. Lifecycle is the executor (one executor per session DO, | ||
| 36 | built in `apps/cloud/src/services/executor.ts#createScopedExecutor`). | ||
| 37 | - LRU with `capacity: 64`, `timeToLive: 5 min`. | ||
| 38 | - Lookup runs inside a Scope held by `cacheScope`, so the pooled | ||
| 39 | connection survives `Effect.scoped` boundaries inside `invokeMcpTool`. | ||
| 40 | - Cache key (`invoke.ts#connectionCacheKey`) is | ||
| 41 | `remote:${invokerScope}:${endpoint}` for remote sources and | ||
| 42 | `stdio:${command}` for stdio. Includes `invokerScope` so per-user | ||
| 43 | OAuth/header secrets never collapse onto a shared connection. | ||
| 44 | - `invokeMcpTool` retries once on connection error: invalidates the | ||
| 45 | cache entry and re-acquires. Stale connections are caught here and | ||
| 46 | transparently re-handshaked. | ||
| 47 | |||
| 48 | ## Why the pool is plausibly correct | ||
| 49 | |||
| 50 | `packages/plugins/mcp/src/sdk/elicitation.test.ts` already had a test | ||
| 51 | `"connection is reused across multiple tool calls to the same source"` | ||
| 52 | that asserts the underlying MCP server sees no new HTTP sessions across | ||
| 53 | three sequential `executor.tools.invoke` calls — passes against current | ||
| 54 | code. | ||
| 55 | |||
| 56 | This PR adds two stricter regression cases under | ||
| 57 | `packages/plugins/mcp/src/sdk/connection-pool.test.ts`: | ||
| 58 | |||
| 59 | 1. Five sequential invokes of the same tool — one handshake total. | ||
| 60 | 2. Different tools on the same source — still one handshake, different | ||
| 61 | tool ids hit the same cache key. | ||
| 62 | |||
| 63 | Both pass without any source change. They serve as a deterministic | ||
| 64 | contract pin so a future refactor that breaks pooling (e.g., scoping | ||
| 65 | the cache to the per-invoke scope, dropping `Scope.extend(cacheScope)`, | ||
| 66 | mutating the cache key per call) trips immediately in CI. | ||
| 67 | |||
| 68 | ## Re-reading the production trace | ||
| 69 | |||
| 70 | Trace `b7102047bed975da461c0519d1251de4`: | ||
| 71 | - `mcp.plugin.resolve_connector` 2.72s | ||
| 72 | - `plugin.mcp.connection.acquire` 2.72s | ||
| 73 | - `plugin.mcp.connection.handshake` 2.52s | ||
| 74 | - `executor.storage.transaction` 1.02s (token persist) | ||
| 75 | - `plugin.mcp.client.call_tool` 1.48s | ||
| 76 | |||
| 77 | Span nesting plus `resolve_connector ≈ acquire ≈ handshake` durations is | ||
| 78 | the cache-miss path: `resolveConnector` only runs (and emits its span) | ||
| 79 | when the cache lookup actually invokes the lookup closure, which only | ||
| 80 | happens on a miss. | ||
| 81 | |||
| 82 | 8h aggregate: | ||
| 83 | - `plugin.mcp.connection.handshake` count 4 | ||
| 84 | - `plugin.mcp.connection.acquire` count 24 | ||
| 85 | - `mcp.plugin.resolve_connector` count 4 | ||
| 86 | |||
| 87 | Acquire count is 6× handshake count. The cache *is* hitting on 20 of 24 | ||
| 88 | calls. The 4 misses are cold-start sessions (each MCP session DO's first | ||
| 89 | tool call, plus any session where the connection went stale and was | ||
| 90 | invalidated by the retry path). Six tool calls per session is consistent | ||
| 91 | with normal MCP usage. | ||
| 92 | |||
| 93 | ## Why we can't drive misses below the cold-start floor | ||
| 94 | |||
| 95 | Every miss observed in prod is structurally unavoidable inside a single | ||
| 96 | DO: | ||
| 97 | |||
| 98 | - **First tool invocation in a fresh DO.** `init()` builds the executor | ||
| 99 | and a fresh `ScopedCache`. The first invoke must handshake. | ||
| 100 | - **Stale-connection retry.** `invokeMcpTool` invalidates the cache | ||
| 101 | entry on `client.callTool` failure and re-acquires; that re-acquire | ||
| 102 | is necessarily a miss. | ||
| 103 | - **TTL expiry.** Set to 5 minutes. The session DO's idle alarm fires | ||
| 104 | at 5 minutes too, so any cache entry that out-survives a session was | ||
| 105 | going to be discarded with the DO anyway. | ||
| 106 | |||
| 107 | To go below this floor we'd need either: | ||
| 108 | |||
| 109 | 1. **Pre-warm in `init()`.** Plausible but speculative — we don't know | ||
| 110 | which sources the user will invoke, and warming all of them | ||
| 111 | bottlenecks `init()` on the slowest server. Out of scope; would also | ||
| 112 | reverse the savings if the user never calls those tools. | ||
| 113 | 2. **Cross-DO connection pool.** Forbidden by the task — and the | ||
| 114 | per-DO scope of `runtimeRef` is the right home for a per-user-org | ||
| 115 | connection pool given the cache key includes `invokerScope`. | ||
| 116 | |||
| 117 | ## What this PR ships | ||
| 118 | |||
| 119 | 1. `packages/plugins/mcp/src/sdk/connection-pool.test.ts` — two | ||
| 120 | strict regression tests pinning the per-session pooling contract. | ||
| 121 | 2. `packages/plugins/mcp/src/sdk/invoke.ts` — adds | ||
| 122 | `plugin.mcp.cache_hit: boolean` attribute to the | ||
| 123 | `plugin.mcp.connection.acquire` span. Future Axiom queries can read | ||
| 124 | the hit rate directly without comparing acquire vs handshake counts. | ||
| 125 | |||
| 126 | No behavior change. | ||
| 127 | |||
| 128 | ## Follow-ups | ||
| 129 | |||
| 130 | - If the cold-start p99 still bites tail latency, consider warming the | ||
| 131 | most-recently-used source(s) for a session DO during `init()` — | ||
| 132 | scoped to the user's recent activity, e.g., on session restore. | ||
| 133 | - Watch `plugin.mcp.cache_hit=false` rate over a longer window. If it | ||
| 134 | exceeds the implied cold-start floor (≈ 1 / avg-calls-per-session), | ||
| 135 | there's a real bug to chase. | ||