Sebastian ThielAdd a plan that consolidates all performance TODOs2facc6f6 months agoHistory
282 lines · 20.7 KB · MarkdownRawDownload

Performance Plan

Imported on: 2026-04-30

Fully generated as overview, intiial motivation is to keep track of gix-index performance and possible improvements.

Source basis:

  • All TODO(perf) markers in this checkout.
  • The two TODO(performance) markers, as they are equivalent in intent.
  • Nearby generic TODOs that may affect common Git performance paths.
  • A local comparison against /Users/byron/dev/github.com/git/git.

Working assumption: priorities are based on ordinary interactive and networked Git use, not on synthetic microbenchmarks alone.

Typical Workflows

These workflows define the impact ordering:

  • git status, git add, and editor integrations that repeatedly refresh the index, classify the worktree, and scan untracked paths.
  • git commit, git checkout, git switch, git merge, and git reset --hard, which depend on fast index reads, tree/index conversion, checkout, and content merge.
  • git diff, git log -- <path>, and git blame, which traverse trees, objects, and diffs repeatedly.
  • git fetch, git pull, and git clone, which stress negotiation, object lookup, pack ingestion, duplicate avoidance, and pack/index refresh.
  • git push, git gc, git repack, and object verification, which stress pack traversal, delta reconstruction, bitmaps, and multi-pack metadata.
  • git archive and export-style commands, which stream trees with attributes and filters.

Priority Backlog

P0: Status, Add, And Index Refresh

  • index-lookup-accelerator: Make index path lookup acceleration a default fast path, not only an ignore-case helper.

    • Source TODOs:
      • gix-dir/src/walk/classify.rs:396: build a multi-threaded hash table so lookups are always accelerated, even for case-sensitive paths.
      • crate-status.md:844: always use multi-threaded initialization of the case-insensitive hash table to accelerate index lookups.
      • gix-index/src/access/mod.rs:144: multi-threaded insertion needs a raw table with multiple bucket locks.
    • Workflow impact: very high for status, add, checkout collision checks, untracked filtering, and pathspec-heavy commands in large worktrees.
    • Git comparison: Git's name-hash.c keeps index_state.name_hash and dir_hash for index lookups. It uses lazy initialization, a per-thread threshold (LAZY_THREAD_COST), bucket-derived locks, a two-phase directory/name hash build, and exact-then-ignore-case comparison. Git's read-cache.c wires name-hash updates into index entry insertion.
    • Plan:
      • Factor gix-index::AccelerateLookup into a general name/directory lookup accelerator usable for case-sensitive and case-insensitive callers.
      • Add a parallel build path with sharded insertion or per-thread partial tables merged at the end.
      • Benchmark status and dirwalk on large repos with core.ignoreCase true and false.

  • status-refresh-heuristics: Close the status refresh heuristic gap.

    • Source TODOs:
      • gix-status/src/index_as_worktree/function.rs:144: decide when parallelization is not worth it; Git uses about 500 entries per thread, capped at 20 preload threads.
      • src/plumbing/main.rs:433: make thread-limit tuning configurable; macOS and Linux scale differently.
      • gix/src/status/mod.rs:165: make Repository::is_dirty() a dedicated early-stop implementation with parallelism.
    • Workflow impact: very high for prompts, IDEs, git status, and dirty checks in automation.
    • Git comparison: Git's preload-index.c uses MAX_PARALLEL = 20, THREAD_COST = 500, skips staged/submodule/up-to-date/skip-worktree/fsmonitor-valid entries, and marks entries up to date after lstat.
    • Plan:
      • Measure the current chunk heuristic against Git-like thresholds on Linux and macOS.
      • Add a dedicated is_dirty() path that stops after the first tree/index or index/worktree change.
      • Make thread policy configurable through repository/config options rather than command-specific constants.

  • untracked-fsmonitor-cache: Use untracked-cache and fsmonitor data aggressively where valid.

    • Source TODOs:
      • crate-status.md:846: accelerated walk with the UNTR extension.
      • gix-index/src/extension/untracked_cache.rs:29 and :33: understand directory stat data and extension semantics fully.
      • gix-status/src/index_as_worktree/traits.rs:105: make streaming I/O interruptible.
    • Workflow impact: very high for status in large repositories with many ignored/untracked files.
    • Git comparison: Git's wt-status.c passes istate->untracked into fill_directory() and avoids full scans when valid; read-cache.c also tweaks fsmonitor and untracked-cache state after index load.
    • Plan:
      • Finish documenting and validating UNTR semantics.
      • Integrate valid untracked-cache state into gix-dir walks and status collection.
      • Preserve interruptibility for editor integrations and long-running worktree scans.

P0: Index Read, Decode, And Tree-To-Index

  • index-decode-storage: Fix index decode storage costs before adding more threads.

    • Source TODOs:
      • gix-index/src/decode/entries.rs:185: path_backing.extend_from_slice() causes large memmove time despite apparent capacity.
      • crate-status.md:856: threaded index read spends most time storing paths and currently has little benefit.
      • gix-index/src/decode/entries.rs:118: entries behave like an intrusive path-keyed collection; this likely affects ignore-case and lookup.
      • gix-index/src/entry/flags.rs:11: use persisted path length to save in-memory entry size.
    • Workflow impact: very high for every command that opens a large repository.
    • Git comparison: Git's read-cache.c memory-maps the index, allocates cache entries from mempools, uses the EOIE and IEOT extensions for threaded extension and entry loading, and stores path bytes inline in struct cache_entry (name[FLEX_ARRAY]). It also records an index entry offset table when threaded reads are requested.
    • Plan:
      • Profile the path_backing growth path and replace it with either pre-sized storage, chunked storage, or per-thread path arenas merged without repeated moves.
      • Evaluate an entry representation that stores path length compactly and gives lookup structures stable references.
      • Treat parallel decode as second-stage work after path storage stops dominating.

  • tree-index-sort: Avoid post-sort work when initializing an index from a tree.

    • Source TODO:
      • gix-index/src/init.rs:107: remove_file_directory_conflicts() sorts only to protect against invalid trees; typical valid trees already compare in order.
    • Workflow impact: high for checkout, archive setup, worktree stream, and operations that synthesize indexes from trees.
    • Git comparison: Git's cache-tree machinery (cache-tree.c) validates tree/index order and can prime an index's cache tree directly from a tree. read-cache.c and cache-tree.c keep sorted index invariants central.
    • Plan:
      • Insert entries in sorted order when walking trees and detect invalid file/directory conflicts during insertion.
      • Add a fast path for already sorted valid trees and a fall-back verifier for malformed data.
      • Reuse or extend cache-tree information when constructing an index from HEAD.

  • worktree-stream-traversal: Reduce double traversal in worktree streaming and archive creation.

    • Source TODOs:
      • gix/src/repository/worktree.rs:88: use the index at HEAD if possible.
      • gix/src/repository/worktree.rs:89: non-HEAD trees are effectively traversed twice; object-cache sharing across copied ODB handles is not trivial.
    • Workflow impact: medium to high for archive, export, and tooling that streams a tree with attributes.
    • Git comparison: Git relies heavily on cache-tree and index state to avoid redundant tree work when the requested tree matches index/HEAD state.
    • Plan:
      • Fast-path HEAD through an existing index plus cache-tree where available.
      • For non-HEAD trees, evaluate an explicit shared object/header cache for the two traversal consumers.

P0: Object Database And Pack Access

  • odb-refresh-wait: Replace spin/yield waiting during dynamic pack-index refresh.

    • Source TODO:
      • gix-odb/src/store_impls/dynamic/load_index.rs:152: a potentially hot loop should probably be a condition variable.
    • Workflow impact: high for fetch, clone, object lookup under concurrency, and commands that observe newly written packs.
    • Git comparison: Git's pack store preparation is centralized in packfile.c, and pack loading/re-preparation is explicit. It avoids leaving multiple callers in a spin loop while another caller is loading the same pack metadata.
    • Plan:
      • Replace the yield loop with a wait/notify primitive around index-load state transitions.
      • Measure concurrent object lookup during fetch and after pack creation.

  • loose-object-single-lookup: Avoid duplicate loose-object lookup work.

    • Source TODOs:
      • gix-odb/src/store_impls/dynamic/find.rs:292: remove loose DB contains() plus try_find() double lookup.
      • gix-odb/src/store_impls/dynamic/header.rs:166: same double lookup for headers.
    • Workflow impact: medium to high for loose-heavy repositories, freshly created objects, and test fixtures.
    • Git comparison: Git's ODB lookup API returns object info/content from one lookup path and then moves packs toward the front after hits.
    • Plan:
      • Add a single-call loose-object API that can report missing, header-only, or full object data.
      • Preserve the current borrow-safety shape by returning owned metadata where needed.

  • pack-delta-base-cache: Revisit pack delta reconstruction and base caching.

    • Source TODO:
      • gix-pack/src/data/file/decode/entry.rs:381: optimize memory-intensive delta-chain reconstruction after more tests exist.
    • Workflow impact: high for checkout after fetch, clone verification, diff/log over packed history, and blame.
    • Git comparison: Git's packfile.c uses a delta_base_cache keyed by pack and base offset with an LRU size limit, a small preallocated delta stack, and delayed insertion into the cache to avoid races while unpacking.
    • Plan:
      • Add coverage for long and shared delta chains first.
      • Compare current buffer swapping with Git's base-cache strategy and adopt an LRU base cache if repeated base materialization is visible in profiles.

  • fetch-duplicate-objects: Investigate duplicate received objects during fetch.

    • Source TODO:
      • gix/tests/gix/remote/fetch.rs:128: tests observe substantial duplication when receiving objects.
    • Workflow impact: high for fetch, pull, and clone.
    • Git comparison: Git uses bitmaps, multi-pack-index, pack reuse, and object-entry de-duplication in pack-object paths.
    • Plan:
      • Determine if duplication is transport negotiation, pack ingestion, replacement refs, or iteration/reporting.
      • Add a counter-based regression test around unique received object IDs and pack refresh count.

P1: Diff, Blame, Merge, And History Queries

  • blame-rename-diff-cache: Avoid duplicated tree diff work in blame rename detection.

    • Source TODO:
      • gix-blame/src/file/function.rs:582: rename tracking repeats tree diff work after the no-rewrite pass.
    • Workflow impact: medium to high for git blame, especially with -M, -C, and path history over rename-heavy repositories.
    • Git comparison: Git's blame.c first looks for unchanged origins, then renames, then optionally moves/copies. It carries blame_origin, blame_entry, score thresholds, and optional blame Bloom data rather than blindly repeating the same tree walk for every case.
    • Plan:
      • Feed the first path-limited tree diff into a rewrite tracker when an addition or deletion is found.
      • Cache parent/current tree IDs and blob headers per commit pair.
      • Evaluate a blame-specific origin cache modeled after Git's two-pass origin/rename structure.

  • merge-conflict-structures: Audit merge tree conflict data structures.

    • Source TODO:
      • gix-merge/src/tree/mod.rs:376: a better data structure may be needed for some directory/file conflict representation.
    • Workflow impact: medium for merges and rebases in large trees.
    • Git comparison: Git's merge and rename machinery keeps maps for directory rename counts and per-path pairing in diffcore-rename.c.
    • Plan:
      • Profile rename and directory/file conflict heavy merges.
      • Replace ad hoc per-conflict representation only if lookup or rewrite costs dominate.

  • diff-preprocess-rescans: Remove avoidable rescans in diff preprocessing.

    • Source TODO:
      • gix-imara-diff/src/myers/preprocess.rs:132: do not unnecessarily rescan lines.
    • Workflow impact: medium for large file diffs and blame.
    • Git comparison: Git's xdiff stack has dedicated preprocessing and histogram/patience variants. gix-imara-diff already carries Git-inspired histogram behavior, so this should remain benchmark-led.
    • Plan:
      • Add benchmarks for common source-file diffs and large generated files.
      • Cache sliding-window counts or use prefix/suffix pruning data already computed during tokenization.

P1: Repository Discovery, Refs, And Path Handling

  • packed-ref-lookup: Implement packed-buffer aware reference lookup in general handles.

    • Source TODO:
      • gix-ref/src/store/general/handle/find.rs:28: implement lookup with packed-buffer handling.
    • Workflow impact: medium for commands that resolve many refs, such as branch listing, fetch negotiation, and revision parsing.
    • Git comparison: Git's ref backend keeps packed refs as a searchable table and layers loose refs on top.
    • Plan:
      • Add a packed-ref buffer search path with loose-over-packed precedence.
      • Benchmark ref-heavy repos with many tags.

  • literal-path-normalization: Avoid path normalization through pattern machinery for single literal paths.

    • Source TODOs:
      • gitoxide-core/src/repository/blame.rs:29, gitoxide-core/src/repository/merge/file.rs:37, tests/it/src/commands/blame_copy_royal.rs:58: normalize paths without going through patterns.
    • Workflow impact: medium for porcelain commands over individual paths.
    • Plan:
      • Add a literal path normalization helper below pathspec parsing.
      • Keep pathspec matching for real pattern inputs only.

P2: Allocation And Micro-Hotspots

  • hex-prefix-stack-decode: Remove heap allocation from odd-length hex prefix parsing.

    • Source TODO:
      • gix-hash/src/prefix.rs:120: decode odd hex prefixes without heap allocation.
    • Workflow impact: low to medium, but may matter for abbreviated object lookup and ref/path disambiguation.
    • Git comparison: Git's object abbreviation logic in object-name.c and packfile.c works against fixed-size object IDs and pack fanout searches rather than allocating per prefix.
    • Plan:
      • Decode into a stack buffer sized by Kind::longest() and copy directly into ObjectId.
      • Add a microbenchmark for odd and even prefix parsing.

  • small-attribute-values: Use a small byte-string representation for attribute values.

    • Source TODO:
      • gix-attributes/src/state.rs:8: a small byte string could provide an estimated 5 percent improvement.
    • Workflow impact: low to medium for status, archive, checkout, and filter-heavy repos.
    • Plan:
      • Prototype a smallvec-backed byte string with display/serde wrappers.
      • Validate JSON behavior and attribute query benchmarks before adopting.

  • signed-data-reader: Stream signed commit data without allocation.

    • Source TODO:
      • gix-object/src/commit/mod.rs:33: implement std::io::Read for SignedData.
    • Workflow impact: low to medium, mostly signature verification and tooling over many signed commits.
    • Plan:
      • Add a segmented reader over the two non-signature ranges instead of building a BString.

  • packetline-borrow-decode: Remove extra packet-line decoding caused by borrow-checker workarounds.

    • Source TODO:
      • gix-packetline/src/blocking_io/read.rs:110: avoid additional decoding of the internal buffer.
    • Workflow impact: low to medium for fetch/clone protocol handling.
    • Plan:
      • Revisit API lifetimes and return owned or borrowed line data explicitly.

  • tempfile-registry-map: Re-evaluate tempfile registry maps.

    • Source TODO:
      • gix-tempfile/src/lib.rs:75: use a gix-hashtable slot-map once available.
    • Workflow impact: low for typical workflows, higher for checkout/merge paths that create many temporary files.
    • Plan:
      • Benchmark the existing mutexed map against a slot-map and the hp-hashmap feature.

  • git-date-token-parser: Replace brute-force Git date parsing.

    • Source TODO:
      • gix-date/src/parse/git.rs:11: learn from Git's parser instead of generated brute-force parsing.
    • Workflow impact: low for normal operations, medium for log/rev parsing over many date filters.
    • Git comparison: Git's date.c uses a token scanner (parse_date_basic) with alpha, digit, and timezone matchers, then falls back to approxidate.
    • Plan:
      • Replace format-chain parsing with a token scanner for Git-specific formats.
      • Keep fixtures generated from Git for compatibility.

Git-Inspired Candidates Without Current TODOs

  • sparse-index-parity: Sparse-index parity for worktree-heavy commands.

    • Git comparison: Git's sparse-index.c can collapse full index ranges into sparse directory entries and expand on demand.
    • Candidate impact: very high for monorepos and sparse checkouts.
    • Next step: audit which gix-status, gix-dir, checkout, and merge code paths force full-index behavior.

  • commit-graph-bloom: Commit-graph generation and Bloom-filter use in history walks.

    • Git comparison: Git's revision.c uses commit-graph generation numbers to bound traversal, and blame can initialize Bloom data.
    • Candidate impact: high for log -- <path>, merge-base, blame, and negotiation.
    • Next step: verify all revision walkers prefer generation numbers and changed-path filters when available.

  • midx-revindex-bitmaps: Multi-pack-index reverse-index and bitmap-driven object enumeration.

    • Git comparison: Git's midx.c, pack-revindex.c, and packfile.c use MIDX lookup, reverse-index chunks, and prefixed-object iteration across MIDX before falling back to individual packs.
    • Candidate impact: high for fetch, clone, gc, prefix lookup, and object iteration in repos with many packs.
    • Next step: compare gix-pack and gix-odb MIDX support against Git's lookup order and reverse-index cache behavior.

  • pack-reuse-delta-policy: Pack reuse and path-based delta compression policies.

    • Git comparison: Git's builtin/pack-objects.c uses pack reuse, bitmap reuse, delta islands, path-based regions, threaded delta search, and delta-cache limits.
    • Candidate impact: high for push, fetch serving, gc, and repack.
    • Next step: treat this as a separate pack-generation roadmap after read-side pack performance is measured.

Measurement Plan

  • Use Git's GIT_TRACE_PERFORMANCE=true and GIT_TRACE2_PERF=1 baselines on the same repositories as gix measurements.
  • Maintain at least these benchmark repositories: small everyday repo, git/git, Linux-sized history, a many-untracked-files worktree, a sparse checkout, and a many-pack repository.
  • Track wall time, syscalls/stat count, object reads, pack index refreshes, peak RSS, and allocation counts.
  • For each P0 item, require one benchmark that represents an interactive command and one regression test that protects the intended fast path.

Immediate Next Moves

  • profile-status-worktrees: Profile gix status on large case-sensitive and case-insensitive worktrees and compare against Git's name-hash/preload behavior.
  • profile-index-decode: Profile index decode to identify why path_backing triggers memmove, then choose arena/chunked storage before expanding threaded decode.
  • implement-odb-wait-notify: Replace the dynamic ODB pack-index load wait loop with wait/notify.
  • fetch-dup-counter: Add a fetch duplicate-object diagnostic counter before changing negotiation or pack ingestion.
  • prototype-prefix-stack: Prototype stack-only odd-prefix parsing as a low-risk starter change.

Exit Criteria

  • status-scale-exit: status and dirty checks scale with index size and untracked-file count within an agreed multiplier of Git on the benchmark corpus.
  • index-throughput-exit: Index read throughput improves on large indexes without increasing peak memory disproportionately.
  • object-lookup-wait-exit: Concurrent object lookup during fetch/clone no longer spins while another thread loads pack metadata.
  • pack-benchmark-coverage-exit: Object lookup and pack delta reconstruction have benchmark coverage before cache/data-structure rewrites land.
  • todo-triage-exit: The remaining performance TODOs are either implemented, benchmarked and rejected, or converted into narrower tracked tasks.