History

Add a plan that consolidates all performance TODOs

Sebastian Thiel committed 6 months ago

Commit 2facc6f1 parent f3cc6b91 file · +282 −0Browse files

Unified · Split

etc/plan/performance.md+282−0
@@ -0,0 +1,282 @@
1+# Performance Plan
2+
3+Imported on: 2026-04-30
4+
5+Fully generated as overview, intiial motivation is to keep track of `gix-index` performance and possible improvements.
6+
7+Source basis:
8+
9+- All `TODO(perf)` markers in this checkout.
10+- The two `TODO(performance)` markers, as they are equivalent in intent.
11+- Nearby generic TODOs that may affect common Git performance paths.
12+- A local comparison against `/Users/byron/dev/github.com/git/git`.
13+
14+Working assumption: priorities are based on ordinary interactive and networked Git use, not on synthetic microbenchmarks alone.
15+
16+## Typical Workflows
17+
18+These workflows define the impact ordering:
19+
20+- `git status`, `git add`, and editor integrations that repeatedly refresh the index, classify the worktree, and scan untracked paths.
21+- `git commit`, `git checkout`, `git switch`, `git merge`, and `git reset --hard`, which depend on fast index reads, tree/index conversion, checkout, and content merge.
22+- `git diff`, `git log -- <path>`, and `git blame`, which traverse trees, objects, and diffs repeatedly.
23+- `git fetch`, `git pull`, and `git clone`, which stress negotiation, object lookup, pack ingestion, duplicate avoidance, and pack/index refresh.
24+- `git push`, `git gc`, `git repack`, and object verification, which stress pack traversal, delta reconstruction, bitmaps, and multi-pack metadata.
25+- `git archive` and export-style commands, which stream trees with attributes and filters.
26+
27+## Priority Backlog
28+
29+### P0: Status, Add, And Index Refresh
30+
31+- [ ] **index-lookup-accelerator**: Make index path lookup acceleration a default fast path, not only an ignore-case helper.
32+ - Source TODOs:
33+ - `gix-dir/src/walk/classify.rs:396`: build a multi-threaded hash table so lookups are always accelerated, even for case-sensitive paths.
34+ - `crate-status.md:844`: always use multi-threaded initialization of the case-insensitive hash table to accelerate index lookups.
35+ - `gix-index/src/access/mod.rs:144`: multi-threaded insertion needs a raw table with multiple bucket locks.
36+ - Workflow impact: very high for `status`, `add`, checkout collision checks, untracked filtering, and pathspec-heavy commands in large worktrees.
37+ - Git comparison: Git's `name-hash.c` keeps `index_state.name_hash` and `dir_hash` for index lookups. It uses lazy initialization, a per-thread threshold (`LAZY_THREAD_COST`), bucket-derived locks, a two-phase directory/name hash build, and exact-then-ignore-case comparison. Git's `read-cache.c` wires name-hash updates into index entry insertion.
38+ - Plan:
39+ - Factor `gix-index::AccelerateLookup` into a general name/directory lookup accelerator usable for case-sensitive and case-insensitive callers.
40+ - Add a parallel build path with sharded insertion or per-thread partial tables merged at the end.
41+ - Benchmark `status` and `dirwalk` on large repos with `core.ignoreCase` true and false.
42+
43+- [ ] **status-refresh-heuristics**: Close the status refresh heuristic gap.
44+ - Source TODOs:
45+ - `gix-status/src/index_as_worktree/function.rs:144`: decide when parallelization is not worth it; Git uses about 500 entries per thread, capped at 20 preload threads.
46+ - `src/plumbing/main.rs:433`: make thread-limit tuning configurable; macOS and Linux scale differently.
47+ - `gix/src/status/mod.rs:165`: make `Repository::is_dirty()` a dedicated early-stop implementation with parallelism.
48+ - Workflow impact: very high for prompts, IDEs, `git status`, and dirty checks in automation.
49+ - Git comparison: Git's `preload-index.c` uses `MAX_PARALLEL = 20`, `THREAD_COST = 500`, skips staged/submodule/up-to-date/skip-worktree/fsmonitor-valid entries, and marks entries up to date after `lstat`.
50+ - Plan:
51+ - Measure the current chunk heuristic against Git-like thresholds on Linux and macOS.
52+ - Add a dedicated `is_dirty()` path that stops after the first tree/index or index/worktree change.
53+ - Make thread policy configurable through repository/config options rather than command-specific constants.
54+
55+- [ ] **untracked-fsmonitor-cache**: Use untracked-cache and fsmonitor data aggressively where valid.
56+ - Source TODOs:
57+ - `crate-status.md:846`: accelerated walk with the `UNTR` extension.
58+ - `gix-index/src/extension/untracked_cache.rs:29` and `:33`: understand directory stat data and extension semantics fully.
59+ - `gix-status/src/index_as_worktree/traits.rs:105`: make streaming I/O interruptible.
60+ - Workflow impact: very high for `status` in large repositories with many ignored/untracked files.
61+ - Git comparison: Git's `wt-status.c` passes `istate->untracked` into `fill_directory()` and avoids full scans when valid; `read-cache.c` also tweaks fsmonitor and untracked-cache state after index load.
62+ - Plan:
63+ - Finish documenting and validating `UNTR` semantics.
64+ - Integrate valid untracked-cache state into `gix-dir` walks and status collection.
65+ - Preserve interruptibility for editor integrations and long-running worktree scans.
66+
67+### P0: Index Read, Decode, And Tree-To-Index
68+
69+- [ ] **index-decode-storage**: Fix index decode storage costs before adding more threads.
70+ - Source TODOs:
71+ - `gix-index/src/decode/entries.rs:185`: `path_backing.extend_from_slice()` causes large `memmove` time despite apparent capacity.
72+ - `crate-status.md:856`: threaded index read spends most time storing paths and currently has little benefit.
73+ - `gix-index/src/decode/entries.rs:118`: entries behave like an intrusive path-keyed collection; this likely affects ignore-case and lookup.
74+ - `gix-index/src/entry/flags.rs:11`: use persisted path length to save in-memory entry size.
75+ - Workflow impact: very high for every command that opens a large repository.
76+ - Git comparison: Git's `read-cache.c` memory-maps the index, allocates cache entries from mempools, uses the EOIE and IEOT extensions for threaded extension and entry loading, and stores path bytes inline in `struct cache_entry` (`name[FLEX_ARRAY]`). It also records an index entry offset table when threaded reads are requested.
77+ - Plan:
78+ - Profile the `path_backing` growth path and replace it with either pre-sized storage, chunked storage, or per-thread path arenas merged without repeated moves.
79+ - Evaluate an entry representation that stores path length compactly and gives lookup structures stable references.
80+ - Treat parallel decode as second-stage work after path storage stops dominating.
81+
82+- [ ] **tree-index-sort**: Avoid post-sort work when initializing an index from a tree.
83+ - Source TODO:
84+ - `gix-index/src/init.rs:107`: `remove_file_directory_conflicts()` sorts only to protect against invalid trees; typical valid trees already compare in order.
85+ - Workflow impact: high for checkout, archive setup, worktree stream, and operations that synthesize indexes from trees.
86+ - Git comparison: Git's cache-tree machinery (`cache-tree.c`) validates tree/index order and can prime an index's cache tree directly from a tree. `read-cache.c` and `cache-tree.c` keep sorted index invariants central.
87+ - Plan:
88+ - Insert entries in sorted order when walking trees and detect invalid file/directory conflicts during insertion.
89+ - Add a fast path for already sorted valid trees and a fall-back verifier for malformed data.
90+ - Reuse or extend cache-tree information when constructing an index from `HEAD`.
91+
92+- [ ] **worktree-stream-traversal**: Reduce double traversal in worktree streaming and archive creation.
93+ - Source TODOs:
94+ - `gix/src/repository/worktree.rs:88`: use the index at `HEAD` if possible.
95+ - `gix/src/repository/worktree.rs:89`: non-HEAD trees are effectively traversed twice; object-cache sharing across copied ODB handles is not trivial.
96+ - Workflow impact: medium to high for `archive`, export, and tooling that streams a tree with attributes.
97+ - Git comparison: Git relies heavily on cache-tree and index state to avoid redundant tree work when the requested tree matches index/HEAD state.
98+ - Plan:
99+ - Fast-path `HEAD` through an existing index plus cache-tree where available.
100+ - For non-HEAD trees, evaluate an explicit shared object/header cache for the two traversal consumers.
101+
102+### P0: Object Database And Pack Access
103+
104+- [ ] **odb-refresh-wait**: Replace spin/yield waiting during dynamic pack-index refresh.
105+ - Source TODO:
106+ - `gix-odb/src/store_impls/dynamic/load_index.rs:152`: a potentially hot loop should probably be a condition variable.
107+ - Workflow impact: high for fetch, clone, object lookup under concurrency, and commands that observe newly written packs.
108+ - Git comparison: Git's pack store preparation is centralized in `packfile.c`, and pack loading/re-preparation is explicit. It avoids leaving multiple callers in a spin loop while another caller is loading the same pack metadata.
109+ - Plan:
110+ - Replace the yield loop with a wait/notify primitive around index-load state transitions.
111+ - Measure concurrent object lookup during fetch and after pack creation.
112+
113+- [ ] **loose-object-single-lookup**: Avoid duplicate loose-object lookup work.
114+ - Source TODOs:
115+ - `gix-odb/src/store_impls/dynamic/find.rs:292`: remove loose DB `contains()` plus `try_find()` double lookup.
116+ - `gix-odb/src/store_impls/dynamic/header.rs:166`: same double lookup for headers.
117+ - Workflow impact: medium to high for loose-heavy repositories, freshly created objects, and test fixtures.
118+ - Git comparison: Git's ODB lookup API returns object info/content from one lookup path and then moves packs toward the front after hits.
119+ - Plan:
120+ - Add a single-call loose-object API that can report missing, header-only, or full object data.
121+ - Preserve the current borrow-safety shape by returning owned metadata where needed.
122+
123+- [ ] **pack-delta-base-cache**: Revisit pack delta reconstruction and base caching.
124+ - Source TODO:
125+ - `gix-pack/src/data/file/decode/entry.rs:381`: optimize memory-intensive delta-chain reconstruction after more tests exist.
126+ - Workflow impact: high for checkout after fetch, clone verification, diff/log over packed history, and blame.
127+ - Git comparison: Git's `packfile.c` uses a `delta_base_cache` keyed by pack and base offset with an LRU size limit, a small preallocated delta stack, and delayed insertion into the cache to avoid races while unpacking.
128+ - Plan:
129+ - Add coverage for long and shared delta chains first.
130+ - Compare current buffer swapping with Git's base-cache strategy and adopt an LRU base cache if repeated base materialization is visible in profiles.
131+
132+- [ ] **fetch-duplicate-objects**: Investigate duplicate received objects during fetch.
133+ - Source TODO:
134+ - `gix/tests/gix/remote/fetch.rs:128`: tests observe substantial duplication when receiving objects.
135+ - Workflow impact: high for fetch, pull, and clone.
136+ - Git comparison: Git uses bitmaps, multi-pack-index, pack reuse, and object-entry de-duplication in pack-object paths.
137+ - Plan:
138+ - Determine if duplication is transport negotiation, pack ingestion, replacement refs, or iteration/reporting.
139+ - Add a counter-based regression test around unique received object IDs and pack refresh count.
140+
141+### P1: Diff, Blame, Merge, And History Queries
142+
143+- [ ] **blame-rename-diff-cache**: Avoid duplicated tree diff work in blame rename detection.
144+ - Source TODO:
145+ - `gix-blame/src/file/function.rs:582`: rename tracking repeats tree diff work after the no-rewrite pass.
146+ - Workflow impact: medium to high for `git blame`, especially with `-M`, `-C`, and path history over rename-heavy repositories.
147+ - Git comparison: Git's `blame.c` first looks for unchanged origins, then renames, then optionally moves/copies. It carries `blame_origin`, `blame_entry`, score thresholds, and optional blame Bloom data rather than blindly repeating the same tree walk for every case.
148+ - Plan:
149+ - Feed the first path-limited tree diff into a rewrite tracker when an addition or deletion is found.
150+ - Cache parent/current tree IDs and blob headers per commit pair.
151+ - Evaluate a blame-specific origin cache modeled after Git's two-pass origin/rename structure.
152+
153+- [ ] **merge-conflict-structures**: Audit merge tree conflict data structures.
154+ - Source TODO:
155+ - `gix-merge/src/tree/mod.rs:376`: a better data structure may be needed for some directory/file conflict representation.
156+ - Workflow impact: medium for merges and rebases in large trees.
157+ - Git comparison: Git's merge and rename machinery keeps maps for directory rename counts and per-path pairing in `diffcore-rename.c`.
158+ - Plan:
159+ - Profile rename and directory/file conflict heavy merges.
160+ - Replace ad hoc per-conflict representation only if lookup or rewrite costs dominate.
161+
162+- [ ] **diff-preprocess-rescans**: Remove avoidable rescans in diff preprocessing.
163+ - Source TODO:
164+ - `gix-imara-diff/src/myers/preprocess.rs:132`: do not unnecessarily rescan lines.
165+ - Workflow impact: medium for large file diffs and blame.
166+ - Git comparison: Git's xdiff stack has dedicated preprocessing and histogram/patience variants. `gix-imara-diff` already carries Git-inspired histogram behavior, so this should remain benchmark-led.
167+ - Plan:
168+ - Add benchmarks for common source-file diffs and large generated files.
169+ - Cache sliding-window counts or use prefix/suffix pruning data already computed during tokenization.
170+
171+### P1: Repository Discovery, Refs, And Path Handling
172+
173+- [ ] **packed-ref-lookup**: Implement packed-buffer aware reference lookup in general handles.
174+ - Source TODO:
175+ - `gix-ref/src/store/general/handle/find.rs:28`: implement lookup with packed-buffer handling.
176+ - Workflow impact: medium for commands that resolve many refs, such as branch listing, fetch negotiation, and revision parsing.
177+ - Git comparison: Git's ref backend keeps packed refs as a searchable table and layers loose refs on top.
178+ - Plan:
179+ - Add a packed-ref buffer search path with loose-over-packed precedence.
180+ - Benchmark ref-heavy repos with many tags.
181+
182+- [ ] **literal-path-normalization**: Avoid path normalization through pattern machinery for single literal paths.
183+ - Source TODOs:
184+ - `gitoxide-core/src/repository/blame.rs:29`, `gitoxide-core/src/repository/merge/file.rs:37`, `tests/it/src/commands/blame_copy_royal.rs:58`: normalize paths without going through patterns.
185+ - Workflow impact: medium for porcelain commands over individual paths.
186+ - Plan:
187+ - Add a literal path normalization helper below pathspec parsing.
188+ - Keep pathspec matching for real pattern inputs only.
189+
190+### P2: Allocation And Micro-Hotspots
191+
192+- [ ] **hex-prefix-stack-decode**: Remove heap allocation from odd-length hex prefix parsing.
193+ - Source TODO:
194+ - `gix-hash/src/prefix.rs:120`: decode odd hex prefixes without heap allocation.
195+ - Workflow impact: low to medium, but may matter for abbreviated object lookup and ref/path disambiguation.
196+ - Git comparison: Git's object abbreviation logic in `object-name.c` and `packfile.c` works against fixed-size object IDs and pack fanout searches rather than allocating per prefix.
197+ - Plan:
198+ - Decode into a stack buffer sized by `Kind::longest()` and copy directly into `ObjectId`.
199+ - Add a microbenchmark for odd and even prefix parsing.
200+
201+- [ ] **small-attribute-values**: Use a small byte-string representation for attribute values.
202+ - Source TODO:
203+ - `gix-attributes/src/state.rs:8`: a small byte string could provide an estimated 5 percent improvement.
204+ - Workflow impact: low to medium for status, archive, checkout, and filter-heavy repos.
205+ - Plan:
206+ - Prototype a `smallvec`-backed byte string with display/serde wrappers.
207+ - Validate JSON behavior and attribute query benchmarks before adopting.
208+
209+- [ ] **signed-data-reader**: Stream signed commit data without allocation.
210+ - Source TODO:
211+ - `gix-object/src/commit/mod.rs:33`: implement `std::io::Read` for `SignedData`.
212+ - Workflow impact: low to medium, mostly signature verification and tooling over many signed commits.
213+ - Plan:
214+ - Add a segmented reader over the two non-signature ranges instead of building a `BString`.
215+
216+- [ ] **packetline-borrow-decode**: Remove extra packet-line decoding caused by borrow-checker workarounds.
217+ - Source TODO:
218+ - `gix-packetline/src/blocking_io/read.rs:110`: avoid additional decoding of the internal buffer.
219+ - Workflow impact: low to medium for fetch/clone protocol handling.
220+ - Plan:
221+ - Revisit API lifetimes and return owned or borrowed line data explicitly.
222+
223+- [ ] **tempfile-registry-map**: Re-evaluate tempfile registry maps.
224+ - Source TODO:
225+ - `gix-tempfile/src/lib.rs:75`: use a `gix-hashtable` slot-map once available.
226+ - Workflow impact: low for typical workflows, higher for checkout/merge paths that create many temporary files.
227+ - Plan:
228+ - Benchmark the existing mutexed map against a slot-map and the `hp-hashmap` feature.
229+
230+- [ ] **git-date-token-parser**: Replace brute-force Git date parsing.
231+ - Source TODO:
232+ - `gix-date/src/parse/git.rs:11`: learn from Git's parser instead of generated brute-force parsing.
233+ - Workflow impact: low for normal operations, medium for log/rev parsing over many date filters.
234+ - Git comparison: Git's `date.c` uses a token scanner (`parse_date_basic`) with alpha, digit, and timezone matchers, then falls back to approxidate.
235+ - Plan:
236+ - Replace format-chain parsing with a token scanner for Git-specific formats.
237+ - Keep fixtures generated from Git for compatibility.
238+
239+## Git-Inspired Candidates Without Current TODOs
240+
241+- [ ] **sparse-index-parity**: Sparse-index parity for worktree-heavy commands.
242+ - Git comparison: Git's `sparse-index.c` can collapse full index ranges into sparse directory entries and expand on demand.
243+ - Candidate impact: very high for monorepos and sparse checkouts.
244+ - Next step: audit which `gix-status`, `gix-dir`, checkout, and merge code paths force full-index behavior.
245+
246+- [ ] **commit-graph-bloom**: Commit-graph generation and Bloom-filter use in history walks.
247+ - Git comparison: Git's `revision.c` uses commit-graph generation numbers to bound traversal, and blame can initialize Bloom data.
248+ - Candidate impact: high for `log -- <path>`, merge-base, blame, and negotiation.
249+ - Next step: verify all revision walkers prefer generation numbers and changed-path filters when available.
250+
251+- [ ] **midx-revindex-bitmaps**: Multi-pack-index reverse-index and bitmap-driven object enumeration.
252+ - Git comparison: Git's `midx.c`, `pack-revindex.c`, and `packfile.c` use MIDX lookup, reverse-index chunks, and prefixed-object iteration across MIDX before falling back to individual packs.
253+ - Candidate impact: high for fetch, clone, gc, prefix lookup, and object iteration in repos with many packs.
254+ - Next step: compare `gix-pack` and `gix-odb` MIDX support against Git's lookup order and reverse-index cache behavior.
255+
256+- [ ] **pack-reuse-delta-policy**: Pack reuse and path-based delta compression policies.
257+ - Git comparison: Git's `builtin/pack-objects.c` uses pack reuse, bitmap reuse, delta islands, path-based regions, threaded delta search, and delta-cache limits.
258+ - Candidate impact: high for push, fetch serving, gc, and repack.
259+ - Next step: treat this as a separate pack-generation roadmap after read-side pack performance is measured.
260+
261+## Measurement Plan
262+
263+- Use Git's `GIT_TRACE_PERFORMANCE=true` and `GIT_TRACE2_PERF=1` baselines on the same repositories as `gix` measurements.
264+- Maintain at least these benchmark repositories: small everyday repo, `git/git`, Linux-sized history, a many-untracked-files worktree, a sparse checkout, and a many-pack repository.
265+- Track wall time, syscalls/stat count, object reads, pack index refreshes, peak RSS, and allocation counts.
266+- For each P0 item, require one benchmark that represents an interactive command and one regression test that protects the intended fast path.
267+
268+## Immediate Next Moves
269+
270+- [ ] **profile-status-worktrees**: Profile `gix status` on large case-sensitive and case-insensitive worktrees and compare against Git's name-hash/preload behavior.
271+- [ ] **profile-index-decode**: Profile index decode to identify why `path_backing` triggers `memmove`, then choose arena/chunked storage before expanding threaded decode.
272+- [ ] **implement-odb-wait-notify**: Replace the dynamic ODB pack-index load wait loop with wait/notify.
273+- [ ] **fetch-dup-counter**: Add a fetch duplicate-object diagnostic counter before changing negotiation or pack ingestion.
274+- [ ] **prototype-prefix-stack**: Prototype stack-only odd-prefix parsing as a low-risk starter change.
275+
276+## Exit Criteria
277+
278+- [ ] **status-scale-exit**: `status` and dirty checks scale with index size and untracked-file count within an agreed multiplier of Git on the benchmark corpus.
279+- [ ] **index-throughput-exit**: Index read throughput improves on large indexes without increasing peak memory disproportionately.
280+- [ ] **object-lookup-wait-exit**: Concurrent object lookup during fetch/clone no longer spins while another thread loads pack metadata.
281+- [ ] **pack-benchmark-coverage-exit**: Object lookup and pack delta reconstruction have benchmark coverage before cache/data-structure rewrites land.
282+- [ ] **todo-triage-exit**: The remaining performance TODOs are either implemented, benchmarked and rejected, or converted into narrower tracked tasks.