The cloud e2e stack degrades after a few minutes of sustained suite load and eventually hangs, which is the CI cascade flake: MCP-heavy shards (toolkits-mcp et al) run, then unrelated specs die with Playwright TimeoutErrors because requests hang rather than erroring. Root cause is postgres connection churn against the dev PGlite server, not SSE frame buffering or OTel span growth. Two compounding problems: - The dev-db PGlite socket server (apps/cloud/scripts/dev-db.ts) used the library default maxConnections: 1. The cloud worker opens a fresh postgres pool per request (the MCP auth seam rebuilds one on every /mcp request to respect workerd's per-request I/O rule), so under concurrent load the second-and-later connects were answered with "Too many connections" and an immediate socket close. postgres.js then reconnected in a tight loop, piling up thousands of half-closed sockets, starving real queries and driving request latency into the tens of seconds until the stack hung. PGlite runs queries serially anyway (its QueryQueueManager uses runExclusive), so allowing many connections is safe: they queue instead of storming. Raise the cap (overridable via DEV_DB_MAX_CONNECTIONS). - The postgres pool finalizer (apps/cloud/src/db/db.ts) was fire-and-forget: Effect.runFork(sql.end({ timeout: 0 })) returned before the socket was torn down, so closing sockets accumulated faster than the server reaped them. Await a graceful end() inside the acquireRelease scope so the number of live-plus-closing sockets stays bounded to what is actually in flight. Extracted as closePostgres() and covered by a unit test. Measured on a booted cloud stack (3 concurrent MCP clients, 1s think-time): baseline (maxConnections=1): ~18 sessions in 135s, avg 15s/session, p95 up to 50s, latency climbing, stack stalling, thousands of "Too many connections". fixed: 347 sessions in 135s, avg ~125ms/session, p95 ~150ms, flat, zero "Too many connections".
Showing the changes to apps/cloud/scripts (1 of 3 files). Show all
Unified · Split
| @@ -82,10 +82,28 @@ const db = await PGlite.create(DB_PATH); · 81 unmodified lines | ||
| 82 | 82 | console.log(`[dev-db] Running migrations from ${MIGRATIONS_FOLDER}`); |
| 83 | 83 | await migrate(drizzle(db), { migrationsFolder: MIGRATIONS_FOLDER }); |
| 84 | 84 | |
| 85 | +// `PGLiteSocketServer` defaults to `maxConnections: 1` and answers every extra | |
| 86 | +// concurrent connection with "Too many connections" + an immediate socket | |
| 87 | +// close. (pglite-socket 0.1.4's published index.d.ts documents "default: 100", | |
| 88 | +// but the shipped runtime JS is `maxConnections ?? 1`, verified in the shipped | |
| 89 | +// chunk, so the runtime default really is 1.) The cloud worker opens a fresh | |
| 90 | +// postgres pool per request (the MCP auth seam rebuilds one on EVERY `/mcp` | |
| 91 | +// request, see apps/cloud/src/mcp/auth.ts), so under concurrent load, exactly | |
| 92 | +// what the e2e suite generates against one shared dev stack, the | |
| 93 | +// second-and-later connects were rejected, and postgres.js reconnected in a | |
| 94 | +// tight loop against the closed socket. That reconnect storm piled up | |
| 95 | +// thousands of half-closed sockets, starved real queries, drove request | |
| 96 | +// latency into the tens of seconds, and eventually hung the stack: the CI e2e | |
| 97 | +// "cloud dev stack degrades after minutes of sustained load" cascade flake. | |
| 98 | +// PGlite still runs queries serially (its internal QueryQueueManager executes | |
| 99 | +// each under `runExclusive`), so allowing many connections is safe: they queue | |
| 100 | +// instead of being rejected. Raise the cap so concurrent requests wait their | |
| 101 | +// turn instead of storming. | |
| 85 | 102 | const server = new PGLiteSocketServer({ |
| 86 | 103 | db, |
| 87 | 104 | port: PORT, |
| 88 | 105 | host: "127.0.0.1", |
| 106 | + maxConnections: Number(process.env.DEV_DB_MAX_CONNECTIONS ?? 1000), | |
| 89 | 107 | }); |
| 90 | 108 | |
| 91 | 109 | await server.start(); |