gitoxide

gix is a command-line interface (CLI) to access git repositories in various ways best described as low-level for use by experts or those validating functionality in real-world scenarios. Performance and efficiency are staples of the implementation.

ein is reserved for one-off tools that are useful to many, and will one day implement a truly unique workflow with the potential to become the preferred way to interact with git repositories.

Please note that all functionality comes from the gitoxide-core library, which mirrors these capabilities and itself relies on all gix-* crates. It's not meant for consumption, for application development, please use gix.

• the ein program - convenient and for humans
  • init - initialize a new non-bare repository with a main branch
  • clone - initialize a local copy of a remote repository
  • tools
    • organize - find all git repositories and place them in directories according to their remote paths
    • find - find all git repositories in a given directory - useful for tools like skim
    • estimate-hours - estimate the time invested into a repository by evaluating commit dates.
      • Based on the git-hours algorithm.
      • See the discussion for some performance data.
• the gix program (plumbing) - lower level commands for use during development
  • As its main purpose is to help running the latest improvements in the real world, it's self-documenting without duplicating its features here. Use gix --help to start discovery.

Plumbing Workflow Overview

This overview intentionally ignores gix CLI availability. It tracks whether the plumbing exists to implement a typical Git command in an application or a future porcelain.

Items from the historical gix 1.0 tracking issue are folded into the relevant crate sections below if they still match today's code and goals.

Note: Generated by Codex, but still good for an overview. Reconcile on demand.

• workflows already supported in plumbing
  • clone, fetch, ls-refs and shallow variants
  • commit and low-level ref/object/index mutation
  • status and worktree-vs-index comparison
  • blob and tree diffing - mainly gix-diff and gix
  • merge-base, revision parsing and commit description
  • worktree streaming, archive creation and low-level checkout
• common workflows that still need plumbing across crates
  • checkout, switch, restore and reset - mainly gix, gix-worktree-state, gix-index, gix-filter, gix-hook
  • merge, cherry-pick and revert - mainly gix, gix-merge, gix-cherry-pick, gix-sequencer, gix-rerere, gix-hook
  • rebase - mainly gix-rebase, gix-sequencer, gix-merge, gix-apply, gix-rerere, gix-hook, gix
  • bisect - mainly gix-bisect, gix-sequencer, gix-revision, gix-revwalk, gix-traverse, gix
  • stash, am and apply - mainly gix-stash, gix-apply, gix-mailbox, gix, gix-diff, gix-worktree-state, gix-sequencer, gix-rerere
  • push and self-contained clone/fetch over file:// and ssh:// - mainly gix, gix-protocol, gix-transport, gix-ssh
• cross-cutting parity work
  • hook discovery and execution - mainly gix-hook, gix, gix-ref, gix-protocol
  • Git 3.0 compatibility (SHA-256, reftable) - mainly gix-hash, gix-ref, gix-reftable, gix-protocol, gix-odb
  • partial clone, promisor and bundle bootstrapping - mainly gix-odb, gix-pack, gix-bundle, gix-protocol, gix
  • big-repo accelerators (commit-graph, bitmaps, split-index, sparse-index, fsmonitor, untracked-cache) - mainly gix-commitgraph, gix-pack, gix-index, gix-status, gix-dir
  • incremental ancestry exclusion during commit iteration - mainly gix-revwalk, gix-traverse, gix-revision, gix-commitgraph

gix

The top-level crate that acts as hub to all functionality provided by the gix-* plumbing crates.

• utilities for applications to make long running operations interruptible gracefully and to support timeouts in servers.
• handle core.repositoryFormatVersion and extensions
• support for unicode-precomposition of command-line arguments (needs explicit use in parent application)
• strict object creation (validate objects referenced by newly created objects exist)
• strict hash verification (validate that objects actually have the hashes they claim to have)
• workflow composition
  • checkout, switch, restore and reset orchestration over refs, index and worktree mutation
  • merge workflow orchestration
    • persist and resume conflicted merges with MERGE_HEAD, MERGE_MSG and MERGE_MODE compatible state
  • rebase workflow orchestration
  • cherry-pick and revert workflow orchestration
  • bisect workflow orchestration
  • stash workflow orchestration
  • git am and git apply workflow orchestration
    • connect mailbox ingestion, patch application, hook execution and resulting commit creation
• Repository
  • discovery
    • option to not cross file systems (default)
    • handle git-common-dir
    • support for GIT_CEILING_DIRECTORIES environment variable
    • handle other non-discovery modes and provide control over environment variable usage required in applications
  • rev-parse
    • handle relative paths as relative to working directory
    • handle upstream and push resolution.
  • rev-walk
    • include tips
    • exclude commits
  • instantiation
  • access to refs and objects
  • create a pathspec-search from a set of strings
    • allow to construct Pathspecs using data structure instead of enforcing them to be passed as strings.
  • credentials
    • run git credential directly
    • use credential helper configuration and to obtain credentials with gix_credentials::helper::Cascade
  • traverse
    • commit graphs
    • make git-notes accessible
    • tree entries
  • diffs/changes
    • tree with other tree
      • respect case-sensitivity of host filesystem.
      • a way to access various diff related settings or use them
      • respect diff.*.textconv, diff.*.cachetextconv and external diff viewers with diff.*.command, along with support for reading diff gitattributes.
      • rewrite tracking
        • deviation - git keeps up to four candidates whereas we use the first-found candidate that matches the similarity percentage. This can lead to different sources being found. As such, we also don't consider the filename at all.
        • handle binary files correctly, and apply filters for that matter
        • computation limit with observable reduction of precision when it is hit, for copies and renames separately
        • by identity
          • renames (sym-links are only ever compared by identity)
          • copies
        • by similarity - similarity factor controllable separately from renames
          • renames
          • copies
        • 'find-copies-harder' - find copies with the source being the entire tree.
    • tree or index with working tree
      • rename tracking
      • submodule status (recursive)
    • diffs between modified blobs with various algorithms
    • tree with index (via index-from-tree and index)
      • rename tracking
      • submodule status (recursive)
  • initialize
    • Proper configuration depending on platform (e.g. ignorecase, filemode, …)
  • Id
    • short hashes with detection of ambiguity.
  • Commit
    • git describe like functionality, with optional commit-graph acceleration
    • create new commit from tree
  • Objects
    • lookup
    • peel to object kind
    • create signed commits and tags
    • trees
      • lookup path
      • edit
  • references
    • peel to end
    • ref-log access
    • remote name
    • find remote itself
      • respect branch.<name>.merge in the returned remote.
  • remotes
    • clone
      • shallow
        • include-tags when shallow is used (needs separate fetch)
        • prune non-existing shallow commits
      • bundles
    • fetch
      • shallow (remains shallow, options to adjust shallow boundary)
      • a way to auto-explode small packs to avoid them to pile up
      • 'ref-in-want'
      • 'wanted-ref'
      • standard negotiation algorithms consecutive, skipping and noop.
    • push
    • ls-refs
    • ls-refs with ref-spec filter
    • list, find by name
    • create in memory
    • save to configuration on disk
    • write FETCH_HEAD
    • apply transport and remote configuration from git-config, including http.*
    • groups
    • remote and branch files
  • execute hooks
    • respect core.hooksPath
    • client-side hooks for checkout, commit, rebase, merge, am and push
    • receive-side hooks and reference-transaction
  • refs
    • run transaction hooks and handle special repository states like quarantine
    • support for different backends like files and reftable
  • main or linked worktree
    • add files with .gitignore handling
    • checkout with conversions like clean + smudge as in .gitattributes
    • diff index with working tree
    • sparse checkout support
    • read per-worktree config if extensions.worktreeConfig is enabled.
    • index
      • tree from index
      • index from tree
  • worktrees
    • open a repository with worktrees
      • read locked state
      • obtain 'prunable' information
    • proper handling of worktree related refs
    • create a byte stream and create archives for such a stream, including worktree filters and conversions
    • create, move, remove, and repair
    • access exclude information
    • access attribute information
    • respect core.worktree configuration
      • deviation
        • The delicate interplay between GIT_COMMON_DIR and GIT_WORK_TREE isn't implemented.
  • config
    • read the primitive types boolean, integer, string
    • read and interpolate trusted paths
    • low-level API for more elaborate access to all details of git-config files
    • a way to make changes to individual configuration files in memory
    • write configuration back
    • auto-refresh configuration values after they changed on disk
    • facilities to apply the url-match algorithm and to normalize urls before comparison.
  • mailmap
  • object replacements (git replace)
  • read git configuration
  • merging
  • stashing
  • Use Commit Graph to speed up certain queries
  • subtree
  • interactive rebase status/manipulation
  • submodules
    • handle 'old' form for reading and detect old form
    • list
    • edit
• API documentation
  • Some examples

gix-actor

• read and write a signature that uniquely identifies an actor within a git repository
• a way to parse name <email> tuples (instead of full signatures) to facilitate parsing commit trailers.
• a way to write only actors, useful for commit trailers.

gix-hash

• types to represent hash digests to identify git objects.
• used to abstract over different kinds of hashes, like SHA1 and the upcoming SHA256
• API documentation
  • Some examples

gix-chunk

• decode the chunk file table of contents and provide convenient API
• write the table of contents

gix-hashtable

• hashmap
• hashset

gix-utils

• filesystem
  • probe capabilities
  • symlink creation and removal
  • file snapshots
• BString Interner with Arena-Backing and arbitrary value association
  • probably based on internment, but needs bumpalo support to avoid item allocations/boxing, and avoid internal Mutex. (key type is pointer based).

gix-fs

• probe capabilities
• symlink creation and removal
• file snapshots
• stack abstraction

gix-object

• decode (zero-copy) borrowed objects
  • commit
    • parse trailers
  • tree
• encode owned objects
  • commit
  • tree
  • tag
    • [name validation][tagname-validation]
• transform borrowed to owned objects
• edit trees efficiently and write changes back
  • See if gix-fs::InternedMap improves performance.
• API documentation
  • Some examples

gix-pack

• packs
  • traverse pack index
  • 'object' abstraction
    • decode (zero copy)
    • verify checksum
  • simple and fast pack traversal
  • decode
    • full objects
    • deltified objects
  • decode
    • decode a pack from Read input
      • Add support for zlib-ng for 20% faster decompression performance
      • Read to Iterator of entries
        • read as is, verify hash, and restore partial packs
    • create index from pack alone (much faster than git)
      • resolve 'thin' packs
  • encode
    • Add support for zlib-ng for 2.5x compression performance
    • objects to entries iterator
      • input objects as-is
      • pack only changed objects as derived from input
      • base object compression
      • delta compression
        • respect the delta=false attribute
      • create 'thin' pack, i.e. deltas that are based on objects the other side has.
      • parallel implementation that scales perfectly
    • entries to pack data iterator
    • write index along with the new pack
  • verify pack with statistics
    • brute force - less memory
    • indexed - optimal speed, but more memory
  • advanced
    • Multi-Pack index file (MIDX)
      • read
      • write
      • verify
    • 'bitmap' file
    • promisor-aware maintenance and repack behavior
    • special handling for networked packs
    • detect and retry packed object reading
• API documentation
  • Some examples

gix-odb

• loose object store
  • traverse
  • read
    • into memory
    • streaming
    • verify checksum
  • streaming write for blobs
  • buffer write for small in-memory objects/non-blobs to bring IO down to open-read-close == 3 syscalls
  • read object header (size + kind) without full decompression
• dynamic store
  • auto-refresh of on-disk state
  • handles alternates
  • multi-pack indices
  • perfect scaling with cores
  • support for pack caches, object caches and MRU for best per-thread performance.
  • prefix/short-id lookup, with optional listing of ambiguous objects.
  • object replacements (git replace)
  • high-speed packed object traversal without wasted CPU time
    • user defined filters
  • read object header (size + kind) without full decompression
• sink
  • write objects and obtain id
• alternates
  • resolve links between object databases
  • safe with cycles and recursive configurations
  • multi-line with comments and quotes
• promisor
  • mark promisor packs and remember their source remote
  • fetch missing objects on demand from promisor remotes
  • support multiple promisor remotes and extensions.partialClone
  • make object lookups, maintenance and connectivity checks promisor-aware
• API documentation
  • Some examples

gix-diff

Check out the performance discussion as well.

• tree
  • changes needed to obtain other tree
• blobs
  • patches
    • There are various ways to generate a patch from two blobs.
    • text
    • binary
    • git-apply compatibility
    • merge hunks that are close enough based on line-setting (interhunk-lines)
    • white-space related settings
  • lines
    • Simple line-by-line diffs powered by the imara-diff crate.
• generic rename tracker to find renames and copies
  • find blobs by exact match
  • find blobs by similarity check
  • heuristics to find best candidate
  • find by basename to support similarity check
    • Not having it can lead to issues when files with the same or similar content are part of a move as files can be lost that way.
  • directory tracking
    • by identity
    • by similarity
• blob
  • a choice of to-worktree, to-git and to-worktree-if-needed conversions
  • textconv filters
  • special handling of files beyond the big-file threshold.
  • detection of binary files by looking at header (first 8k bytes)
  • caching of diff-able data
  • prepare invocation of external diff program
    • pass meta-info
• working with hunks of data
• diff-heuristics match Git perfectly
• API documentation
  • Examples

gix-merge

• three-way content-merge analysis of blobs with choice of how to resolve conflicts
  • respect git attributes and drivers.
  • choose how to resolve conflicts on the data-structure
  • more efficient handling of paths with merge=binary attributes (do not load them into memory)
  • produce a new blob based on data-structure containing possible resolutions
    • merge style
    • diff3 style
    • zdiff style
  • various newlines-related options during the merge (see https://git-scm.com/docs/git-merge#Documentation/git-merge.txt-ignore-space-change).
  • a way to control inter-hunk merging based on proximity (maybe via gix-diff feature which could use the same)
• tree-diff-heuristics match Git for its test-cases
  • a way to generate an index with stages, mostly conforming with Git.
  • resolve to be ours or the ancestors version of the tree.
  • submodule merges (right now they count as conflicts if they differ)
  • assure sparse indices are handled correctly during application - right now we refuse.
  • rewrite so that the whole logic can be proven to be correct - it's too insane now and probably has way more possible states than are tested, despite best attempts.
• commits - with handling of multiple merge bases by recursive merge-base merge
• persist merge-in-progress state compatible with MERGE_HEAD, MERGE_MSG and MERGE_MODE
• support merge continuation and abort flows as used by git merge --continue and git merge --abort
• API documentation
  • Examples

gix-blame

• commit-annotations for a single file
  • progress
  • interruptibility
  • streaming

• support for worktree changes (creates virtual commit on top of HEAD)
• shallow-history support
• rename tracking (track different paths through history)
• commits to ignore
• pass all blame-cornercases (from Git)

• Performance-Improvements
  • Without the following the performance isn't competitive with Git.
  1. Implement custom graph walk which won't run down parents that don't have the path in question.
  2. Implement access of trees from commit-graph and fill that information into the traversal info by default.
  3. commit-graph with bloom filter, used to quickly check if a commit has a path.
• API documentation
  • Examples

gix-traverse

Check out the performance discussion as well.

• trees
  • nested traversal
• commits
  • ancestor graph traversal similar to git revlog
  • (performance) stop descending hidden ancestry early instead of prepainting it all
  • commitgraph support
• API documentation
  • Examples

gix-shallow

A utility crate with types and functionality related to shallow-file handling.

gix-url

• As documented here: https://www.git-scm.com/docs/git-clone#_git_urls
• parse
  • ssh URLs and SCP like syntax
  • file, git, and SSH
  • paths (OS paths, without need for UTF-8)
• username expansion for ssh and git urls
• convert URL to string
• API documentation
  • Some examples

gix-packetline

• PKT-Line
• encode
• decode (zero-copy)
• error line
• V2 additions
• sideband mode
• Read from packet line with (optional) progress support via sidebands
• Write with built-in packet line encoding
• async support
• API documentation
  • Some examples

gix-transport

• No matter what we do here, timeouts must be supported to prevent hanging forever and to make interrupts destructor-safe.
• client
  • general purpose connect(…) for clients
    • file:// launches service application
    • file:// without launching git-upload-pack / git-receive-pack
    • ssh:// launches service application in a remote shell using ssh
    • ssh:// without an external ssh binary
    • git:// establishes a tcp connection to a git daemon
    • http(s):// establishes connections to web server
      • via curl (blocking only)
      • via reqwest (blocking only)
    • pass context for scheme specific configuration, like timeouts
  • git://
    • V1 handshake
      • send values + receive data with sidebands
      • support for receiving 'shallow' refs in case the remote repository is shallow itself (I presume)
        • Since V2 doesn't seem to support that, let's skip this until there is an actual need. No completionist :D
    • V2 handshake
      • send command request, receive response with sideband support
  • http(s)://
    • set identity for basic authentication