- Add README entry for perf benchmarks - Document scenarios, commands, artifacts, and env vars
Unified · Split
| @@ -45,6 +45,10 @@ We are very very early in this project. Expect bugs. · 44 unmodified lines | |||
| 45 | 45 | ||
| 46 | We are not accepting contributions yet. | 46 | We are not accepting contributions yet. |
| 47 | 47 | ||
| 48 | ## Performance benchmarks | ||
| 49 | |||
| 50 | See [docs/perf-benchmarks.md](./docs/perf-benchmarks.md) for the local perf harness, seeded scenarios, artifact format, and the commands for automated and manual benchmark runs. | ||
| 51 | |||
| 48 | ## If you REALLY want to contribute still.... read this first | 52 | ## If you REALLY want to contribute still.... read this first |
| 49 | 53 | ||
| 50 | Read [CONTRIBUTING.md](./CONTRIBUTING.md) before opening an issue or PR. | 54 | Read [CONTRIBUTING.md](./CONTRIBUTING.md) before opening an issue or PR. |
| @@ -0,0 +1,267 @@ | |||
| 1 | # Performance Benchmarks | ||
| 2 | |||
| 3 | This repository has a local performance regression harness for the built web app and built server. It is intentionally separate from the normal unit and browser test suites. | ||
| 4 | |||
| 5 | The benchmark is meant to answer two questions: | ||
| 6 | |||
| 7 | - Does large-thread rendering and thread-to-thread navigation stay snappy? | ||
| 8 | - Does high-frequency websocket event application stay responsive under a realistic built-app flow? | ||
| 9 | |||
| 10 | ## What it runs | ||
| 11 | |||
| 12 | The perf suite does not use Vite dev mode. Each run does this: | ||
| 13 | |||
| 14 | 1. Seeds a real server base dir with deterministic fixture data using the normal event store and projection pipeline. | ||
| 15 | 2. Starts the built `t3` server from `apps/server/dist/bin.mjs`. | ||
| 16 | 3. Serves the built web bundle from `apps/server/dist/client`. | ||
| 17 | 4. Launches Chromium with Playwright. | ||
| 18 | 5. Installs an in-page collector for action timings, long tasks, `requestAnimationFrame` gaps, and mounted timeline row counts. | ||
| 19 | 6. Writes a JSON artifact plus server stdout and stderr logs under `artifacts/perf/`. | ||
| 20 | |||
| 21 | The current implementation is browser-focused, but the harness already reserves a server sampling hook so later server-only CPU and memory benchmarks can share the same scenarios and artifact format. | ||
| 22 | |||
| 23 | ## Main pieces | ||
| 24 | |||
| 25 | - `packages/shared/src/perf/scenarioCatalog.ts` | ||
| 26 | - Shared scenario catalog for both seeded state and live provider traffic. | ||
| 27 | - `apps/server/integration/perf/seedPerfState.ts` | ||
| 28 | - Creates the temporary sqlite-backed base dir using real orchestration events and projections. | ||
| 29 | - `apps/server/src/perf/PerfProviderAdapter.ts` | ||
| 30 | - Perf-only mock provider that emits paced runtime events over the normal provider -> websocket -> client path. | ||
| 31 | - `apps/web/test/perf/appHarness.ts` | ||
| 32 | - Starts the built app, launches Chromium, installs the browser collector, and writes artifacts. | ||
| 33 | - `test/perf/support/browserMetrics.ts` | ||
| 34 | - Collects action durations, long tasks, `requestAnimationFrame` gaps, and mounted row samples inside the page. | ||
| 35 | - `test/perf/support/artifact.ts` | ||
| 36 | - Defines the JSON artifact format and summary math. | ||
| 37 | - `test/perf/support/thresholds.ts` | ||
| 38 | - Local budgets that currently gate the suite. | ||
| 39 | - `test/perf/support/serverSampler.ts` | ||
| 40 | - Extension point for future CPU and memory sampling. The current implementation is `NoopServerSampler`, so `serverMetrics` is `null` today. | ||
| 41 | |||
| 42 | ## Scenarios | ||
| 43 | |||
| 44 | ### `large_threads` | ||
| 45 | |||
| 46 | Seeded app state used by the virtualization benchmark. | ||
| 47 | |||
| 48 | - 1 project | ||
| 49 | - 12 threads total | ||
| 50 | - 2 heavy threads | ||
| 51 | - 1 burst thread | ||
| 52 | - 9 light filler threads | ||
| 53 | |||
| 54 | The two heavy threads currently seed: | ||
| 55 | |||
| 56 | - `1,000` turns | ||
| 57 | - `2,000` messages | ||
| 58 | - periodic worklog/activity rows | ||
| 59 | - periodic proposed-plan rows | ||
| 60 | - frequent checkpoint rows with larger changed-file trees | ||
| 61 | |||
| 62 | This means the timeline is not just alternating short user/assistant text. The heavy threads include the non-message rows that tend to stress grouping, virtualization, and diff-tree rendering. | ||
| 63 | |||
| 64 | ### `burst_base` | ||
| 65 | |||
| 66 | Seeded app state used by the websocket benchmark. | ||
| 67 | |||
| 68 | - 1 burst target thread | ||
| 69 | - 1 navigation thread | ||
| 70 | - 1 filler thread | ||
| 71 | |||
| 72 | This smaller seed keeps the focus on live websocket application while still covering cross-thread background activity. | ||
| 73 | |||
| 74 | ### `dense_assistant_stream` | ||
| 75 | |||
| 76 | Live provider scenario used by the websocket benchmark. | ||
| 77 | |||
| 78 | - runs for about `10` seconds | ||
| 79 | - spans `24` cycles | ||
| 80 | - updates `3` lanes at once: `burst`, `navigation`, and `filler` | ||
| 81 | - interleaves assistant messages with multiple worklog/tool lifecycles | ||
| 82 | |||
| 83 | Each cycle emits, per lane: | ||
| 84 | |||
| 85 | 1. an assistant intro message | ||
| 86 | 2. three worklog/tool command lifecycles with file payloads | ||
| 87 | 3. an assistant followup message | ||
| 88 | |||
| 89 | The message fragments vary in length on purpose so the live stream does not render as uniformly sized chunks. | ||
| 90 | |||
| 91 | ## What each benchmark asserts | ||
| 92 | |||
| 93 | ### Virtualization benchmark | ||
| 94 | |||
| 95 | File: `apps/web/test/perf/virtualization.perf.test.ts` | ||
| 96 | |||
| 97 | This benchmark: | ||
| 98 | |||
| 99 | - opens the large-thread seed on the built app | ||
| 100 | - navigates between the two heavy threads through the real sidebar | ||
| 101 | - measures a warmup switch and then six measured thread switches | ||
| 102 | - samples mounted timeline row counts | ||
| 103 | - scrolls bottom -> top -> bottom to catch jank | ||
| 104 | |||
| 105 | It currently fails if the local budgets in `test/perf/support/thresholds.ts` are exceeded: | ||
| 106 | |||
| 107 | - max mounted timeline rows: `140` | ||
| 108 | - thread switch p50: `250ms` | ||
| 109 | - thread switch p95: `500ms` | ||
| 110 | - max long task: `120ms` | ||
| 111 | - max `requestAnimationFrame` gap: `120ms` | ||
| 112 | |||
| 113 | ### Websocket benchmark | ||
| 114 | |||
| 115 | File: `apps/web/test/perf/websocket-application.perf.test.ts` | ||
| 116 | |||
| 117 | This benchmark: | ||
| 118 | |||
| 119 | - starts the built app with the perf provider enabled | ||
| 120 | - opens the burst thread | ||
| 121 | - sends one real composer message | ||
| 122 | - lets the mock provider emit the live multi-thread websocket burst | ||
| 123 | - switches to another thread during the burst and back again | ||
| 124 | - waits for the sentinel text that marks the end of the scenario | ||
| 125 | |||
| 126 | It currently fails if these budgets are exceeded: | ||
| 127 | |||
| 128 | - burst completion: `14,000ms` | ||
| 129 | - max long task: `120ms` | ||
| 130 | - long tasks over `50ms`: `2` | ||
| 131 | - max `requestAnimationFrame` gap: `120ms` | ||
| 132 | - burst-time thread switches: `500ms` max each | ||
| 133 | |||
| 134 | ## Commands | ||
| 135 | |||
| 136 | ### One-time browser setup | ||
| 137 | |||
| 138 | Install Chromium for the perf suite: | ||
| 139 | |||
| 140 | ```bash | ||
| 141 | cd apps/web | ||
| 142 | bun run test:perf:install | ||
| 143 | ``` | ||
| 144 | |||
| 145 | ### Run the full automated benchmark suite | ||
| 146 | |||
| 147 | From the repo root: | ||
| 148 | |||
| 149 | ```bash | ||
| 150 | bun run test:perf:web | ||
| 151 | ``` | ||
| 152 | |||
| 153 | That command builds `@t3tools/web`, builds `t3`, and then runs the dedicated perf Vitest config. | ||
| 154 | |||
| 155 | ### Re-run the perf suite without rebuilding | ||
| 156 | |||
| 157 | If the built artifacts already exist and you have not changed the built app since the last build: | ||
| 158 | |||
| 159 | ```bash | ||
| 160 | cd apps/web | ||
| 161 | bun run test:perf | ||
| 162 | ``` | ||
| 163 | |||
| 164 | ### Watch the automated run in a live browser | ||
| 165 | |||
| 166 | ```bash | ||
| 167 | T3CODE_PERF_HEADFUL=1 bun run test:perf:web | ||
| 168 | ``` | ||
| 169 | |||
| 170 | If you already have fresh build artifacts: | ||
| 171 | |||
| 172 | ```bash | ||
| 173 | cd apps/web | ||
| 174 | T3CODE_PERF_HEADFUL=1 bun run test:perf | ||
| 175 | ``` | ||
| 176 | |||
| 177 | ### Open the seeded app manually for exploration | ||
| 178 | |||
| 179 | Large-thread virtualization state: | ||
| 180 | |||
| 181 | ```bash | ||
| 182 | bun run perf:open:build -- --scenario large_threads --open | ||
| 183 | ``` | ||
| 184 | |||
| 185 | Websocket burst state: | ||
| 186 | |||
| 187 | ```bash | ||
| 188 | bun run perf:open:build -- --scenario burst_base --provider dense_assistant_stream --open | ||
| 189 | ``` | ||
| 190 | |||
| 191 | If the app is already built, use the faster commands: | ||
| 192 | |||
| 193 | ```bash | ||
| 194 | bun run perf:open -- --scenario large_threads --open | ||
| 195 | bun run perf:open -- --scenario burst_base --provider dense_assistant_stream --open | ||
| 196 | ``` | ||
| 197 | |||
| 198 | For the websocket scenario, open the burst thread and send one message to start the live stream. | ||
| 199 | |||
| 200 | ### Inspect the seeded topology directly | ||
| 201 | |||
| 202 | The seed script prints the generated thread summaries as JSON: | ||
| 203 | |||
| 204 | ```bash | ||
| 205 | bun run apps/server/scripts/seedPerfState.ts large_threads | ||
| 206 | bun run apps/server/scripts/seedPerfState.ts burst_base | ||
| 207 | ``` | ||
| 208 | |||
| 209 | ## Artifacts | ||
| 210 | |||
| 211 | By default each run writes to: | ||
| 212 | |||
| 213 | ```text | ||
| 214 | artifacts/perf/<suite>-<scenario>-<timestamp>/ | ||
| 215 | ``` | ||
| 216 | |||
| 217 | Each run currently includes: | ||
| 218 | |||
| 219 | - `<suite>-<scenario>.json` | ||
| 220 | - `<suite>-<scenario>.server.stdout.log` | ||
| 221 | - `<suite>-<scenario>.server.stderr.log` | ||
| 222 | |||
| 223 | The JSON artifact contains: | ||
| 224 | |||
| 225 | - run metadata | ||
| 226 | - threshold profile | ||
| 227 | - summary metrics | ||
| 228 | - raw browser metrics | ||
| 229 | - `serverMetrics`, reserved for later server sampling | ||
| 230 | |||
| 231 | Current summary fields: | ||
| 232 | |||
| 233 | - `maxMountedTimelineRows` | ||
| 234 | - `threadSwitchP50Ms` | ||
| 235 | - `threadSwitchP95Ms` | ||
| 236 | - `maxLongTaskMs` | ||
| 237 | - `longTasksOver50Ms` | ||
| 238 | - `maxRafGapMs` | ||
| 239 | - `burstCompletionMs` | ||
| 240 | |||
| 241 | To change the artifact output directory for one run: | ||
| 242 | |||
| 243 | ```bash | ||
| 244 | T3CODE_PERF_ARTIFACT_DIR=/tmp/t3-perf bun run test:perf:web | ||
| 245 | ``` | ||
| 246 | |||
| 247 | ## Internal env vars | ||
| 248 | |||
| 249 | These are the perf-specific env vars in the current harness: | ||
| 250 | |||
| 251 | - `T3CODE_PERF_HEADFUL=1` | ||
| 252 | - Launch Chromium headed instead of headless. | ||
| 253 | - `T3CODE_PERF_ARTIFACT_DIR=/path/to/output` | ||
| 254 | - Override the artifact directory. | ||
| 255 | - `T3CODE_PERF_PROVIDER=1` | ||
| 256 | - Enables the perf provider path on the server. | ||
| 257 | - `T3CODE_PERF_SCENARIO=dense_assistant_stream` | ||
| 258 | - Selects the live perf provider scenario. | ||
| 259 | |||
| 260 | In normal usage, the automated harness and `perf:open` script set the provider env vars for you. | ||
| 261 | |||
| 262 | ## Notes and limitations | ||
| 263 | |||
| 264 | - This is a local benchmark suite. It is not wired into CI yet. | ||
| 265 | - The normal `apps/web` test suite excludes `test/perf/**/*.perf.test.ts`. Perf tests only run through `apps/web/vitest.perf.config.ts`. | ||
| 266 | - The budgets are intentionally conservative first-pass tripwires, not tuned production SLOs. | ||
| 267 | - The current harness measures browser responsiveness only. Server sampling is an explicit extension point, not implemented metrics yet. | ||