History

Document local performance benchmark harness

- Add README entry for perf benchmarks - Document scenarios, commands, artifacts, and env vars

Julius Marminge committed 7 months ago

Commit b8fba181 parent f1f234d2 files · +271 −0Browse files

Unified · Split

2 files
  1. README.md +4 −0
  2. docs/perf-benchmarks.md +267 −0
README.md+4−0
@@ -45,6 +45,10 @@ We are very very early in this project. Expect bugs. · 44 unmodified lines
4545
4646 We are not accepting contributions yet.
4747
48+## Performance benchmarks
49+
50+See [docs/perf-benchmarks.md](./docs/perf-benchmarks.md) for the local perf harness, seeded scenarios, artifact format, and the commands for automated and manual benchmark runs.
51+
4852 ## If you REALLY want to contribute still.... read this first
4953
5054 Read [CONTRIBUTING.md](./CONTRIBUTING.md) before opening an issue or PR.
docs/perf-benchmarks.md+267−0
@@ -0,0 +1,267 @@
1+# Performance Benchmarks
2+
3+This repository has a local performance regression harness for the built web app and built server. It is intentionally separate from the normal unit and browser test suites.
4+
5+The benchmark is meant to answer two questions:
6+
7+- Does large-thread rendering and thread-to-thread navigation stay snappy?
8+- Does high-frequency websocket event application stay responsive under a realistic built-app flow?
9+
10+## What it runs
11+
12+The perf suite does not use Vite dev mode. Each run does this:
13+
14+1. Seeds a real server base dir with deterministic fixture data using the normal event store and projection pipeline.
15+2. Starts the built `t3` server from `apps/server/dist/bin.mjs`.
16+3. Serves the built web bundle from `apps/server/dist/client`.
17+4. Launches Chromium with Playwright.
18+5. Installs an in-page collector for action timings, long tasks, `requestAnimationFrame` gaps, and mounted timeline row counts.
19+6. Writes a JSON artifact plus server stdout and stderr logs under `artifacts/perf/`.
20+
21+The current implementation is browser-focused, but the harness already reserves a server sampling hook so later server-only CPU and memory benchmarks can share the same scenarios and artifact format.
22+
23+## Main pieces
24+
25+- `packages/shared/src/perf/scenarioCatalog.ts`
26+ - Shared scenario catalog for both seeded state and live provider traffic.
27+- `apps/server/integration/perf/seedPerfState.ts`
28+ - Creates the temporary sqlite-backed base dir using real orchestration events and projections.
29+- `apps/server/src/perf/PerfProviderAdapter.ts`
30+ - Perf-only mock provider that emits paced runtime events over the normal provider -> websocket -> client path.
31+- `apps/web/test/perf/appHarness.ts`
32+ - Starts the built app, launches Chromium, installs the browser collector, and writes artifacts.
33+- `test/perf/support/browserMetrics.ts`
34+ - Collects action durations, long tasks, `requestAnimationFrame` gaps, and mounted row samples inside the page.
35+- `test/perf/support/artifact.ts`
36+ - Defines the JSON artifact format and summary math.
37+- `test/perf/support/thresholds.ts`
38+ - Local budgets that currently gate the suite.
39+- `test/perf/support/serverSampler.ts`
40+ - Extension point for future CPU and memory sampling. The current implementation is `NoopServerSampler`, so `serverMetrics` is `null` today.
41+
42+## Scenarios
43+
44+### `large_threads`
45+
46+Seeded app state used by the virtualization benchmark.
47+
48+- 1 project
49+- 12 threads total
50+- 2 heavy threads
51+- 1 burst thread
52+- 9 light filler threads
53+
54+The two heavy threads currently seed:
55+
56+- `1,000` turns
57+- `2,000` messages
58+- periodic worklog/activity rows
59+- periodic proposed-plan rows
60+- frequent checkpoint rows with larger changed-file trees
61+
62+This means the timeline is not just alternating short user/assistant text. The heavy threads include the non-message rows that tend to stress grouping, virtualization, and diff-tree rendering.
63+
64+### `burst_base`
65+
66+Seeded app state used by the websocket benchmark.
67+
68+- 1 burst target thread
69+- 1 navigation thread
70+- 1 filler thread
71+
72+This smaller seed keeps the focus on live websocket application while still covering cross-thread background activity.
73+
74+### `dense_assistant_stream`
75+
76+Live provider scenario used by the websocket benchmark.
77+
78+- runs for about `10` seconds
79+- spans `24` cycles
80+- updates `3` lanes at once: `burst`, `navigation`, and `filler`
81+- interleaves assistant messages with multiple worklog/tool lifecycles
82+
83+Each cycle emits, per lane:
84+
85+1. an assistant intro message
86+2. three worklog/tool command lifecycles with file payloads
87+3. an assistant followup message
88+
89+The message fragments vary in length on purpose so the live stream does not render as uniformly sized chunks.
90+
91+## What each benchmark asserts
92+
93+### Virtualization benchmark
94+
95+File: `apps/web/test/perf/virtualization.perf.test.ts`
96+
97+This benchmark:
98+
99+- opens the large-thread seed on the built app
100+- navigates between the two heavy threads through the real sidebar
101+- measures a warmup switch and then six measured thread switches
102+- samples mounted timeline row counts
103+- scrolls bottom -> top -> bottom to catch jank
104+
105+It currently fails if the local budgets in `test/perf/support/thresholds.ts` are exceeded:
106+
107+- max mounted timeline rows: `140`
108+- thread switch p50: `250ms`
109+- thread switch p95: `500ms`
110+- max long task: `120ms`
111+- max `requestAnimationFrame` gap: `120ms`
112+
113+### Websocket benchmark
114+
115+File: `apps/web/test/perf/websocket-application.perf.test.ts`
116+
117+This benchmark:
118+
119+- starts the built app with the perf provider enabled
120+- opens the burst thread
121+- sends one real composer message
122+- lets the mock provider emit the live multi-thread websocket burst
123+- switches to another thread during the burst and back again
124+- waits for the sentinel text that marks the end of the scenario
125+
126+It currently fails if these budgets are exceeded:
127+
128+- burst completion: `14,000ms`
129+- max long task: `120ms`
130+- long tasks over `50ms`: `2`
131+- max `requestAnimationFrame` gap: `120ms`
132+- burst-time thread switches: `500ms` max each
133+
134+## Commands
135+
136+### One-time browser setup
137+
138+Install Chromium for the perf suite:
139+
140+```bash
141+cd apps/web
142+bun run test:perf:install
143+```
144+
145+### Run the full automated benchmark suite
146+
147+From the repo root:
148+
149+```bash
150+bun run test:perf:web
151+```
152+
153+That command builds `@t3tools/web`, builds `t3`, and then runs the dedicated perf Vitest config.
154+
155+### Re-run the perf suite without rebuilding
156+
157+If the built artifacts already exist and you have not changed the built app since the last build:
158+
159+```bash
160+cd apps/web
161+bun run test:perf
162+```
163+
164+### Watch the automated run in a live browser
165+
166+```bash
167+T3CODE_PERF_HEADFUL=1 bun run test:perf:web
168+```
169+
170+If you already have fresh build artifacts:
171+
172+```bash
173+cd apps/web
174+T3CODE_PERF_HEADFUL=1 bun run test:perf
175+```
176+
177+### Open the seeded app manually for exploration
178+
179+Large-thread virtualization state:
180+
181+```bash
182+bun run perf:open:build -- --scenario large_threads --open
183+```
184+
185+Websocket burst state:
186+
187+```bash
188+bun run perf:open:build -- --scenario burst_base --provider dense_assistant_stream --open
189+```
190+
191+If the app is already built, use the faster commands:
192+
193+```bash
194+bun run perf:open -- --scenario large_threads --open
195+bun run perf:open -- --scenario burst_base --provider dense_assistant_stream --open
196+```
197+
198+For the websocket scenario, open the burst thread and send one message to start the live stream.
199+
200+### Inspect the seeded topology directly
201+
202+The seed script prints the generated thread summaries as JSON:
203+
204+```bash
205+bun run apps/server/scripts/seedPerfState.ts large_threads
206+bun run apps/server/scripts/seedPerfState.ts burst_base
207+```
208+
209+## Artifacts
210+
211+By default each run writes to:
212+
213+```text
214+artifacts/perf/<suite>-<scenario>-<timestamp>/
215+```
216+
217+Each run currently includes:
218+
219+- `<suite>-<scenario>.json`
220+- `<suite>-<scenario>.server.stdout.log`
221+- `<suite>-<scenario>.server.stderr.log`
222+
223+The JSON artifact contains:
224+
225+- run metadata
226+- threshold profile
227+- summary metrics
228+- raw browser metrics
229+- `serverMetrics`, reserved for later server sampling
230+
231+Current summary fields:
232+
233+- `maxMountedTimelineRows`
234+- `threadSwitchP50Ms`
235+- `threadSwitchP95Ms`
236+- `maxLongTaskMs`
237+- `longTasksOver50Ms`
238+- `maxRafGapMs`
239+- `burstCompletionMs`
240+
241+To change the artifact output directory for one run:
242+
243+```bash
244+T3CODE_PERF_ARTIFACT_DIR=/tmp/t3-perf bun run test:perf:web
245+```
246+
247+## Internal env vars
248+
249+These are the perf-specific env vars in the current harness:
250+
251+- `T3CODE_PERF_HEADFUL=1`
252+ - Launch Chromium headed instead of headless.
253+- `T3CODE_PERF_ARTIFACT_DIR=/path/to/output`
254+ - Override the artifact directory.
255+- `T3CODE_PERF_PROVIDER=1`
256+ - Enables the perf provider path on the server.
257+- `T3CODE_PERF_SCENARIO=dense_assistant_stream`
258+ - Selects the live perf provider scenario.
259+
260+In normal usage, the automated harness and `perf:open` script set the provider env vars for you.
261+
262+## Notes and limitations
263+
264+- This is a local benchmark suite. It is not wired into CI yet.
265+- The normal `apps/web` test suite excludes `test/perf/**/*.perf.test.ts`. Perf tests only run through `apps/web/vitest.perf.config.ts`.
266+- The budgets are intentionally conservative first-pass tripwires, not tuned production SLOs.
267+- The current harness measures browser responsiveness only. Server sampling is an explicit extension point, not implemented metrics yet.