History

Stabilize flaky host-selfhost integration tests

These suites each boot a full self-host app graph and drive it over the in-memory handler; scope-isolation additionally fired 48 unbounded concurrent requests through the single shared libSQL connection. Run in parallel forks sized to the core count, many heavy boots oversubscribe the CPU and starve the event loop, so an in-flight request stalls and the test times out. Which file loses the race is nondeterministic, which is what showed up as CI flakiness. Bound both sources of concurrency rather than leaning on timing: - vitest.config: cap the fork pool at 2 and raise the test/hook timeouts to 30s, so the package stays parallel but never oversubscribes regardless of how many cores the machine has (a many-core dev box oversubscribes harder than a 4-vCPU CI runner). - scope-isolation: read the 6 identities back in bounded rounds (6 in flight, repeated) instead of a 48-wide burst. Same 48 reads and the same disjointness assertions, without piling 48 fibers onto the one serialized connection. Verified green across repeated full-suite runs under heavy local load.

Rhys Sullivan committed 4 months ago

Commit afa6a481 parent 510eb0a2 files · +53 −12Browse files

Unified · Split

2 files
  1. apps/host-selfhost/src/scope-isolation.test.ts +28 −12
  2. apps/host-selfhost/vitest.config.ts +25 −0
apps/host-selfhost/src/scope-isolation.test.ts+28−12
@@ -92,8 +92,17 @@ const listConnectionAddresses = async ( · 91 unmodified lines
9292 };
9393
9494 test("concurrent requests with distinct identities get disjoint, correct executor bindings", async () => {
95− // 6 identities, each its own (org, user). Seed each sequentially, then fire 48
96− // interleaved reads over the one long-lived SQLite handle.
95+ // 6 identities, each its own (org, user). Seed each sequentially, then read
96+ // them back over the one long-lived SQLite handle with all 6 distinct
97+ // identities in flight at once, repeated across rounds.
98+ //
99+ // The leak this guards against is a logic bug (a shared rather than
100+ // request-scoped binding), so it surfaces whenever distinct identities are
101+ // concurrent at all; 6 simultaneous is plenty. We deliberately bound
102+ // in-flight requests to those 6 instead of firing an unbounded 48-wide burst:
103+ // every query serializes through the single libSQL connection, so a 48-fiber
104+ // pile-up starves the event loop under CI load and times out (the flake this
105+ // test was hitting). Same 48 total reads, same assertions, bounded pressure.
97106 const identities = Array.from({ length: 6 }, (_, i) => ({
98107 userId: `user-${i}`,
99108 organizationId: `org-${i}`,
@@ -103,23 +112,30 @@ test("concurrent requests with distinct identities get disjoint, correct executo · 3 unmodified lines
103112 await seedIdentity(id.userId, id.organizationId);
104113 }
105114
106− const requests = Array.from({ length: 48 }, (_, i) => identities[i % identities.length]);
107− const results = await Promise.all(
108− requests.map((id) => listConnectionAddresses(id.userId, id.organizationId)),
109− );
115+ const rounds = 8;
116+ const observed: Array<{ userId: string; status: number; addresses: string[] }> = [];
117+ for (let round = 0; round < rounds; round++) {
118+ const roundResults = await Promise.all(
119+ identities.map(async (id) => ({
120+ userId: id.userId,
121+ ...(await listConnectionAddresses(id.userId, id.organizationId)),
122+ })),
123+ );
124+ observed.push(...roundResults);
125+ }
110126
111− results.forEach((result, i) => {
112− const { userId } = requests[i];
127+ expect(observed).toHaveLength(rounds * identities.length);
128+ observed.forEach(({ userId, status, addresses }) => {
113129 // Each response reflects ONLY its own request's identity — no bleed. The
114130 // subject's own user connection is present, and no OTHER subject's is.
115− expect(result.status).toBe(200);
116− expect(result.addresses.some((a) => a.includes(connectionNameForUser(userId)))).toBe(true);
131+ expect(status).toBe(200);
132+ expect(addresses.some((a) => a.includes(connectionNameForUser(userId)))).toBe(true);
117133 const otherUsers = identities.map((id) => id.userId).filter((u) => u !== userId);
118134 for (const other of otherUsers) {
119− expect(result.addresses.some((a) => a.includes(connectionNameForUser(other)))).toBe(false);
135+ expect(addresses.some((a) => a.includes(connectionNameForUser(other)))).toBe(false);
120136 }
121137 });
122−}, 15_000);
138+}, 30_000);
123139
124140 test("a request with no identity is rejected", async () => {
125141 const res = await handler(new Request("http://localhost/api/connections"));
apps/host-selfhost/vitest.config.ts+25−0
@@ -4,5 +4,30 @@ export default defineConfig({ · 3 unmodified lines
44 test: {
55 include: ["src/**/*.test.ts"],
66 passWithNoTests: true,
7+ // These are integration suites: most boot a full self-host app graph
8+ // (Better Auth + libSQL + MCP + plugins) at module load, then drive it
9+ // over the in-memory handler, and some (scope-isolation) fire dozens of
10+ // concurrent requests through it. Each boot is CPU-heavy and every query
11+ // serializes through the one shared libSQL connection, so when many files
12+ // boot at once they oversubscribe the CPU and starve the event loop: an
13+ // in-flight request stalls indefinitely and the test times out. Which
14+ // files lose the race is nondeterministic, which is the CI flakiness.
15+ //
16+ // Vitest sizes its fork pool to the core count, so a many-core machine
17+ // oversubscribes HARDER than a small CI runner. Bound the concurrency to
18+ // an absolute number instead, so behavior is the same everywhere: at most
19+ // two heavy boots run together, which fits a 4-vCPU runner without
20+ // starvation while still running the suite in parallel. Pair it with
21+ // integration-grade timeouts as headroom for the slowest boot. This keeps
22+ // every assertion intact (no logic or coverage changed) and makes the
23+ // suite deterministic rather than load-dependent.
24+ testTimeout: 30_000,
25+ hookTimeout: 30_000,
26+ poolOptions: {
27+ forks: {
28+ maxForks: 2,
29+ minForks: 1,
30+ },
31+ },
732 },
833 });