History

Add a plan that consolidates all performance TODOs

Sebastian Thiel committed 6 months ago

Commit 2facc6f1 parent f3cc6b91 file · +282 −0Browse files

Unified · Split

etc/plan/performance.md+282−0
@@ -0,0 +1,282 @@
1# Performance Plan
2
3Imported on: 2026-04-30
4
5Fully generated as overview, intiial motivation is to keep track of `gix-index` performance and possible improvements.
6
7Source basis:
8
9- All `TODO(perf)` markers in this checkout.
10- The two `TODO(performance)` markers, as they are equivalent in intent.
11- Nearby generic TODOs that may affect common Git performance paths.
12- A local comparison against `/Users/byron/dev/github.com/git/git`.
13
14Working assumption: priorities are based on ordinary interactive and networked Git use, not on synthetic microbenchmarks alone.
15
16## Typical Workflows
17
18These workflows define the impact ordering:
19
20- `git status`, `git add`, and editor integrations that repeatedly refresh the index, classify the worktree, and scan untracked paths.
21- `git commit`, `git checkout`, `git switch`, `git merge`, and `git reset --hard`, which depend on fast index reads, tree/index conversion, checkout, and content merge.
22- `git diff`, `git log -- <path>`, and `git blame`, which traverse trees, objects, and diffs repeatedly.
23- `git fetch`, `git pull`, and `git clone`, which stress negotiation, object lookup, pack ingestion, duplicate avoidance, and pack/index refresh.
24- `git push`, `git gc`, `git repack`, and object verification, which stress pack traversal, delta reconstruction, bitmaps, and multi-pack metadata.
25- `git archive` and export-style commands, which stream trees with attributes and filters.
26
27## Priority Backlog
28
29### P0: Status, Add, And Index Refresh
30
31- [ ] **index-lookup-accelerator**: Make index path lookup acceleration a default fast path, not only an ignore-case helper.
32 - Source TODOs:
33 - `gix-dir/src/walk/classify.rs:396`: build a multi-threaded hash table so lookups are always accelerated, even for case-sensitive paths.
34 - `crate-status.md:844`: always use multi-threaded initialization of the case-insensitive hash table to accelerate index lookups.
35 - `gix-index/src/access/mod.rs:144`: multi-threaded insertion needs a raw table with multiple bucket locks.
36 - Workflow impact: very high for `status`, `add`, checkout collision checks, untracked filtering, and pathspec-heavy commands in large worktrees.
37 - Git comparison: Git's `name-hash.c` keeps `index_state.name_hash` and `dir_hash` for index lookups. It uses lazy initialization, a per-thread threshold (`LAZY_THREAD_COST`), bucket-derived locks, a two-phase directory/name hash build, and exact-then-ignore-case comparison. Git's `read-cache.c` wires name-hash updates into index entry insertion.
38 - Plan:
39 - Factor `gix-index::AccelerateLookup` into a general name/directory lookup accelerator usable for case-sensitive and case-insensitive callers.
40 - Add a parallel build path with sharded insertion or per-thread partial tables merged at the end.
41 - Benchmark `status` and `dirwalk` on large repos with `core.ignoreCase` true and false.
42
43- [ ] **status-refresh-heuristics**: Close the status refresh heuristic gap.
44 - Source TODOs:
45 - `gix-status/src/index_as_worktree/function.rs:144`: decide when parallelization is not worth it; Git uses about 500 entries per thread, capped at 20 preload threads.
46 - `src/plumbing/main.rs:433`: make thread-limit tuning configurable; macOS and Linux scale differently.
47 - `gix/src/status/mod.rs:165`: make `Repository::is_dirty()` a dedicated early-stop implementation with parallelism.
48 - Workflow impact: very high for prompts, IDEs, `git status`, and dirty checks in automation.
49 - Git comparison: Git's `preload-index.c` uses `MAX_PARALLEL = 20`, `THREAD_COST = 500`, skips staged/submodule/up-to-date/skip-worktree/fsmonitor-valid entries, and marks entries up to date after `lstat`.
50 - Plan:
51 - Measure the current chunk heuristic against Git-like thresholds on Linux and macOS.
52 - Add a dedicated `is_dirty()` path that stops after the first tree/index or index/worktree change.
53 - Make thread policy configurable through repository/config options rather than command-specific constants.
54
55- [ ] **untracked-fsmonitor-cache**: Use untracked-cache and fsmonitor data aggressively where valid.
56 - Source TODOs:
57 - `crate-status.md:846`: accelerated walk with the `UNTR` extension.
58 - `gix-index/src/extension/untracked_cache.rs:29` and `:33`: understand directory stat data and extension semantics fully.
59 - `gix-status/src/index_as_worktree/traits.rs:105`: make streaming I/O interruptible.
60 - Workflow impact: very high for `status` in large repositories with many ignored/untracked files.
61 - Git comparison: Git's `wt-status.c` passes `istate->untracked` into `fill_directory()` and avoids full scans when valid; `read-cache.c` also tweaks fsmonitor and untracked-cache state after index load.
62 - Plan:
63 - Finish documenting and validating `UNTR` semantics.
64 - Integrate valid untracked-cache state into `gix-dir` walks and status collection.
65 - Preserve interruptibility for editor integrations and long-running worktree scans.
66
67### P0: Index Read, Decode, And Tree-To-Index
68
69- [ ] **index-decode-storage**: Fix index decode storage costs before adding more threads.
70 - Source TODOs:
71 - `gix-index/src/decode/entries.rs:185`: `path_backing.extend_from_slice()` causes large `memmove` time despite apparent capacity.
72 - `crate-status.md:856`: threaded index read spends most time storing paths and currently has little benefit.
73 - `gix-index/src/decode/entries.rs:118`: entries behave like an intrusive path-keyed collection; this likely affects ignore-case and lookup.
74 - `gix-index/src/entry/flags.rs:11`: use persisted path length to save in-memory entry size.
75 - Workflow impact: very high for every command that opens a large repository.
76 - Git comparison: Git's `read-cache.c` memory-maps the index, allocates cache entries from mempools, uses the EOIE and IEOT extensions for threaded extension and entry loading, and stores path bytes inline in `struct cache_entry` (`name[FLEX_ARRAY]`). It also records an index entry offset table when threaded reads are requested.
77 - Plan:
78 - Profile the `path_backing` growth path and replace it with either pre-sized storage, chunked storage, or per-thread path arenas merged without repeated moves.
79 - Evaluate an entry representation that stores path length compactly and gives lookup structures stable references.
80 - Treat parallel decode as second-stage work after path storage stops dominating.
81
82- [ ] **tree-index-sort**: Avoid post-sort work when initializing an index from a tree.
83 - Source TODO:
84 - `gix-index/src/init.rs:107`: `remove_file_directory_conflicts()` sorts only to protect against invalid trees; typical valid trees already compare in order.
85 - Workflow impact: high for checkout, archive setup, worktree stream, and operations that synthesize indexes from trees.
86 - Git comparison: Git's cache-tree machinery (`cache-tree.c`) validates tree/index order and can prime an index's cache tree directly from a tree. `read-cache.c` and `cache-tree.c` keep sorted index invariants central.
87 - Plan:
88 - Insert entries in sorted order when walking trees and detect invalid file/directory conflicts during insertion.
89 - Add a fast path for already sorted valid trees and a fall-back verifier for malformed data.
90 - Reuse or extend cache-tree information when constructing an index from `HEAD`.
91
92- [ ] **worktree-stream-traversal**: Reduce double traversal in worktree streaming and archive creation.
93 - Source TODOs:
94 - `gix/src/repository/worktree.rs:88`: use the index at `HEAD` if possible.
95 - `gix/src/repository/worktree.rs:89`: non-HEAD trees are effectively traversed twice; object-cache sharing across copied ODB handles is not trivial.
96 - Workflow impact: medium to high for `archive`, export, and tooling that streams a tree with attributes.
97 - Git comparison: Git relies heavily on cache-tree and index state to avoid redundant tree work when the requested tree matches index/HEAD state.
98 - Plan:
99 - Fast-path `HEAD` through an existing index plus cache-tree where available.
100 - For non-HEAD trees, evaluate an explicit shared object/header cache for the two traversal consumers.
101
102### P0: Object Database And Pack Access
103
104- [ ] **odb-refresh-wait**: Replace spin/yield waiting during dynamic pack-index refresh.
105 - Source TODO:
106 - `gix-odb/src/store_impls/dynamic/load_index.rs:152`: a potentially hot loop should probably be a condition variable.
107 - Workflow impact: high for fetch, clone, object lookup under concurrency, and commands that observe newly written packs.
108 - Git comparison: Git's pack store preparation is centralized in `packfile.c`, and pack loading/re-preparation is explicit. It avoids leaving multiple callers in a spin loop while another caller is loading the same pack metadata.
109 - Plan:
110 - Replace the yield loop with a wait/notify primitive around index-load state transitions.
111 - Measure concurrent object lookup during fetch and after pack creation.
112
113- [ ] **loose-object-single-lookup**: Avoid duplicate loose-object lookup work.
114 - Source TODOs:
115 - `gix-odb/src/store_impls/dynamic/find.rs:292`: remove loose DB `contains()` plus `try_find()` double lookup.
116 - `gix-odb/src/store_impls/dynamic/header.rs:166`: same double lookup for headers.
117 - Workflow impact: medium to high for loose-heavy repositories, freshly created objects, and test fixtures.
118 - Git comparison: Git's ODB lookup API returns object info/content from one lookup path and then moves packs toward the front after hits.
119 - Plan:
120 - Add a single-call loose-object API that can report missing, header-only, or full object data.
121 - Preserve the current borrow-safety shape by returning owned metadata where needed.
122
123- [ ] **pack-delta-base-cache**: Revisit pack delta reconstruction and base caching.
124 - Source TODO:
125 - `gix-pack/src/data/file/decode/entry.rs:381`: optimize memory-intensive delta-chain reconstruction after more tests exist.
126 - Workflow impact: high for checkout after fetch, clone verification, diff/log over packed history, and blame.
127 - Git comparison: Git's `packfile.c` uses a `delta_base_cache` keyed by pack and base offset with an LRU size limit, a small preallocated delta stack, and delayed insertion into the cache to avoid races while unpacking.
128 - Plan:
129 - Add coverage for long and shared delta chains first.
130 - Compare current buffer swapping with Git's base-cache strategy and adopt an LRU base cache if repeated base materialization is visible in profiles.
131
132- [ ] **fetch-duplicate-objects**: Investigate duplicate received objects during fetch.
133 - Source TODO:
134 - `gix/tests/gix/remote/fetch.rs:128`: tests observe substantial duplication when receiving objects.
135 - Workflow impact: high for fetch, pull, and clone.
136 - Git comparison: Git uses bitmaps, multi-pack-index, pack reuse, and object-entry de-duplication in pack-object paths.
137 - Plan:
138 - Determine if duplication is transport negotiation, pack ingestion, replacement refs, or iteration/reporting.
139 - Add a counter-based regression test around unique received object IDs and pack refresh count.
140
141### P1: Diff, Blame, Merge, And History Queries
142
143- [ ] **blame-rename-diff-cache**: Avoid duplicated tree diff work in blame rename detection.
144 - Source TODO:
145 - `gix-blame/src/file/function.rs:582`: rename tracking repeats tree diff work after the no-rewrite pass.
146 - Workflow impact: medium to high for `git blame`, especially with `-M`, `-C`, and path history over rename-heavy repositories.
147 - Git comparison: Git's `blame.c` first looks for unchanged origins, then renames, then optionally moves/copies. It carries `blame_origin`, `blame_entry`, score thresholds, and optional blame Bloom data rather than blindly repeating the same tree walk for every case.
148 - Plan:
149 - Feed the first path-limited tree diff into a rewrite tracker when an addition or deletion is found.
150 - Cache parent/current tree IDs and blob headers per commit pair.
151 - Evaluate a blame-specific origin cache modeled after Git's two-pass origin/rename structure.
152
153- [ ] **merge-conflict-structures**: Audit merge tree conflict data structures.
154 - Source TODO:
155 - `gix-merge/src/tree/mod.rs:376`: a better data structure may be needed for some directory/file conflict representation.
156 - Workflow impact: medium for merges and rebases in large trees.
157 - Git comparison: Git's merge and rename machinery keeps maps for directory rename counts and per-path pairing in `diffcore-rename.c`.
158 - Plan:
159 - Profile rename and directory/file conflict heavy merges.
160 - Replace ad hoc per-conflict representation only if lookup or rewrite costs dominate.
161
162- [ ] **diff-preprocess-rescans**: Remove avoidable rescans in diff preprocessing.
163 - Source TODO:
164 - `gix-imara-diff/src/myers/preprocess.rs:132`: do not unnecessarily rescan lines.
165 - Workflow impact: medium for large file diffs and blame.
166 - Git comparison: Git's xdiff stack has dedicated preprocessing and histogram/patience variants. `gix-imara-diff` already carries Git-inspired histogram behavior, so this should remain benchmark-led.
167 - Plan:
168 - Add benchmarks for common source-file diffs and large generated files.
169 - Cache sliding-window counts or use prefix/suffix pruning data already computed during tokenization.
170
171### P1: Repository Discovery, Refs, And Path Handling
172
173- [ ] **packed-ref-lookup**: Implement packed-buffer aware reference lookup in general handles.
174 - Source TODO:
175 - `gix-ref/src/store/general/handle/find.rs:28`: implement lookup with packed-buffer handling.
176 - Workflow impact: medium for commands that resolve many refs, such as branch listing, fetch negotiation, and revision parsing.
177 - Git comparison: Git's ref backend keeps packed refs as a searchable table and layers loose refs on top.
178 - Plan:
179 - Add a packed-ref buffer search path with loose-over-packed precedence.
180 - Benchmark ref-heavy repos with many tags.
181
182- [ ] **literal-path-normalization**: Avoid path normalization through pattern machinery for single literal paths.
183 - Source TODOs:
184 - `gitoxide-core/src/repository/blame.rs:29`, `gitoxide-core/src/repository/merge/file.rs:37`, `tests/it/src/commands/blame_copy_royal.rs:58`: normalize paths without going through patterns.
185 - Workflow impact: medium for porcelain commands over individual paths.
186 - Plan:
187 - Add a literal path normalization helper below pathspec parsing.
188 - Keep pathspec matching for real pattern inputs only.
189
190### P2: Allocation And Micro-Hotspots
191
192- [ ] **hex-prefix-stack-decode**: Remove heap allocation from odd-length hex prefix parsing.
193 - Source TODO:
194 - `gix-hash/src/prefix.rs:120`: decode odd hex prefixes without heap allocation.
195 - Workflow impact: low to medium, but may matter for abbreviated object lookup and ref/path disambiguation.
196 - Git comparison: Git's object abbreviation logic in `object-name.c` and `packfile.c` works against fixed-size object IDs and pack fanout searches rather than allocating per prefix.
197 - Plan:
198 - Decode into a stack buffer sized by `Kind::longest()` and copy directly into `ObjectId`.
199 - Add a microbenchmark for odd and even prefix parsing.
200
201- [ ] **small-attribute-values**: Use a small byte-string representation for attribute values.
202 - Source TODO:
203 - `gix-attributes/src/state.rs:8`: a small byte string could provide an estimated 5 percent improvement.
204 - Workflow impact: low to medium for status, archive, checkout, and filter-heavy repos.
205 - Plan:
206 - Prototype a `smallvec`-backed byte string with display/serde wrappers.
207 - Validate JSON behavior and attribute query benchmarks before adopting.
208
209- [ ] **signed-data-reader**: Stream signed commit data without allocation.
210 - Source TODO:
211 - `gix-object/src/commit/mod.rs:33`: implement `std::io::Read` for `SignedData`.
212 - Workflow impact: low to medium, mostly signature verification and tooling over many signed commits.
213 - Plan:
214 - Add a segmented reader over the two non-signature ranges instead of building a `BString`.
215
216- [ ] **packetline-borrow-decode**: Remove extra packet-line decoding caused by borrow-checker workarounds.
217 - Source TODO:
218 - `gix-packetline/src/blocking_io/read.rs:110`: avoid additional decoding of the internal buffer.
219 - Workflow impact: low to medium for fetch/clone protocol handling.
220 - Plan:
221 - Revisit API lifetimes and return owned or borrowed line data explicitly.
222
223- [ ] **tempfile-registry-map**: Re-evaluate tempfile registry maps.
224 - Source TODO:
225 - `gix-tempfile/src/lib.rs:75`: use a `gix-hashtable` slot-map once available.
226 - Workflow impact: low for typical workflows, higher for checkout/merge paths that create many temporary files.
227 - Plan:
228 - Benchmark the existing mutexed map against a slot-map and the `hp-hashmap` feature.
229
230- [ ] **git-date-token-parser**: Replace brute-force Git date parsing.
231 - Source TODO:
232 - `gix-date/src/parse/git.rs:11`: learn from Git's parser instead of generated brute-force parsing.
233 - Workflow impact: low for normal operations, medium for log/rev parsing over many date filters.
234 - Git comparison: Git's `date.c` uses a token scanner (`parse_date_basic`) with alpha, digit, and timezone matchers, then falls back to approxidate.
235 - Plan:
236 - Replace format-chain parsing with a token scanner for Git-specific formats.
237 - Keep fixtures generated from Git for compatibility.
238
239## Git-Inspired Candidates Without Current TODOs
240
241- [ ] **sparse-index-parity**: Sparse-index parity for worktree-heavy commands.
242 - Git comparison: Git's `sparse-index.c` can collapse full index ranges into sparse directory entries and expand on demand.
243 - Candidate impact: very high for monorepos and sparse checkouts.
244 - Next step: audit which `gix-status`, `gix-dir`, checkout, and merge code paths force full-index behavior.
245
246- [ ] **commit-graph-bloom**: Commit-graph generation and Bloom-filter use in history walks.
247 - Git comparison: Git's `revision.c` uses commit-graph generation numbers to bound traversal, and blame can initialize Bloom data.
248 - Candidate impact: high for `log -- <path>`, merge-base, blame, and negotiation.
249 - Next step: verify all revision walkers prefer generation numbers and changed-path filters when available.
250
251- [ ] **midx-revindex-bitmaps**: Multi-pack-index reverse-index and bitmap-driven object enumeration.
252 - Git comparison: Git's `midx.c`, `pack-revindex.c`, and `packfile.c` use MIDX lookup, reverse-index chunks, and prefixed-object iteration across MIDX before falling back to individual packs.
253 - Candidate impact: high for fetch, clone, gc, prefix lookup, and object iteration in repos with many packs.
254 - Next step: compare `gix-pack` and `gix-odb` MIDX support against Git's lookup order and reverse-index cache behavior.
255
256- [ ] **pack-reuse-delta-policy**: Pack reuse and path-based delta compression policies.
257 - Git comparison: Git's `builtin/pack-objects.c` uses pack reuse, bitmap reuse, delta islands, path-based regions, threaded delta search, and delta-cache limits.
258 - Candidate impact: high for push, fetch serving, gc, and repack.
259 - Next step: treat this as a separate pack-generation roadmap after read-side pack performance is measured.
260
261## Measurement Plan
262
263- Use Git's `GIT_TRACE_PERFORMANCE=true` and `GIT_TRACE2_PERF=1` baselines on the same repositories as `gix` measurements.
264- Maintain at least these benchmark repositories: small everyday repo, `git/git`, Linux-sized history, a many-untracked-files worktree, a sparse checkout, and a many-pack repository.
265- Track wall time, syscalls/stat count, object reads, pack index refreshes, peak RSS, and allocation counts.
266- For each P0 item, require one benchmark that represents an interactive command and one regression test that protects the intended fast path.
267
268## Immediate Next Moves
269
270- [ ] **profile-status-worktrees**: Profile `gix status` on large case-sensitive and case-insensitive worktrees and compare against Git's name-hash/preload behavior.
271- [ ] **profile-index-decode**: Profile index decode to identify why `path_backing` triggers `memmove`, then choose arena/chunked storage before expanding threaded decode.
272- [ ] **implement-odb-wait-notify**: Replace the dynamic ODB pack-index load wait loop with wait/notify.
273- [ ] **fetch-dup-counter**: Add a fetch duplicate-object diagnostic counter before changing negotiation or pack ingestion.
274- [ ] **prototype-prefix-stack**: Prototype stack-only odd-prefix parsing as a low-risk starter change.
275
276## Exit Criteria
277
278- [ ] **status-scale-exit**: `status` and dirty checks scale with index size and untracked-file count within an agreed multiplier of Git on the benchmark corpus.
279- [ ] **index-throughput-exit**: Index read throughput improves on large indexes without increasing peak memory disproportionately.
280- [ ] **object-lookup-wait-exit**: Concurrent object lookup during fetch/clone no longer spins while another thread loads pack metadata.
281- [ ] **pack-benchmark-coverage-exit**: Object lookup and pack delta reconstruction have benchmark coverage before cache/data-structure rewrites land.
282- [ ] **todo-triage-exit**: The remaining performance TODOs are either implemented, benchmarked and rejected, or converted into narrower tracked tasks.