---

title: Portable stores nav_order: 13 permalink: /portable-stores/

Portable stores

{: .no_toc }

A Git-backed publish target for a gitcrawl.db plus its derived bodies — share a local cache across agents and machines without running a hosted service. {: .fs-6 .fw-300 }

1. TOC {:toc}

When to use one

• You want every agent on a team to read from a shared, recently synced cache without each agent making its own GitHub calls.
• You want a backup of the SQLite cache that someone else can clone and use immediately.
• You want a deterministic snapshot of "what gitcrawl knew at time T" for reproducible triage.

A portable store is just a Git repository whose contents include a SQLite database. Anyone with read access to the repository can git clone it and have a fully populated gitcrawl mirror in seconds.

Setup: pointing gitcrawl at a portable store

gitcrawl init \
  --portable-store https://github.com/openclaw/gitcrawl-store.git \
  --portable-db data/openclaw__openclaw.sync.db

When --store-dir is omitted, gitcrawl clones the store under <config-dir>/stores/<repo-name>. For the example above, a fresh macOS install uses ~/Library/Application Support/gitcrawl/stores/gitcrawl-store; a Linux install using the default config location uses ~/.config/gitcrawl/stores/gitcrawl-store. Pass --store-dir when you want a fixed checkout path.

init will:

1. Clone the portable store next to the active config, or to --store-dir if provided
2. Wire the active config.toml to use the database at --portable-db inside that checkout. With the Linux default config location, that file is ~/.config/gitcrawl/config.toml.
3. Create the runtime cache, vector, and log directories in the standard locations

JSON output reports portable_store_url, portable_store_dir, and portable_store: cloned|pulled|reset-pulled so automation can tell what happened.

Large stores may commit a gzip artifact instead of the raw SQLite file. The configured db_path remains the logical SQLite path, while its sibling manifest declares compression: "gzip", a relative archivePath, and the archive size and SHA-256. Gitcrawl verifies the compressed artifact, inflates at most the manifest's declared SQLite size, verifies the existing uncompressed size and SHA-256, runs SQLite quick_check, and only then atomically replaces the runtime mirror. Legacy raw SQLite stores continue to use the same manifest without compression fields.

How read-only commands behave

Read-only commands (search, threads, clusters, cluster-detail, neighbors, the TUI) refresh the portable-store checkout before reading, so they always see the latest published data:

• The refresh is best-effort and non-interactive
• SSH attempts are bounded so an offline remote does not hang the CLI
• Stale SQLite sidecars (WAL, SHM) are cleared after the pull so queries see freshly pulled data
• Local Git pull configuration that tries to rebase onto multiple branch merge refs is handled cleanly
• SQLite files are copied through a temporary runtime database, checked with quick_check, and verified against the portable manifest before replacing the previous runtime mirror
• Manifest-backed gzip SQLite artifacts are verified before inflation and still use the uncompressed database digest as the runtime identity

If the remote is unreachable, the read still answers from the local checkout.

How write commands behave

Write commands (sync, embed, refresh, portable prune, cluster, neighbor generation) open a writable runtime mirror alongside the portable checkout so new GitHub data, vectors, and overrides persist without partially mutating the published portable store. When this redirect engages, gitcrawl prints one stderr notice naming both the runtime mirror and the checkout database.

JSON output from sync, refresh, fill-pr-details, and portable prune makes that destination machine-readable: db_target is runtime-mirror, db_target_path names the database actually written, and portable_source_db names the database in the checkout. With a non-portable local database, db_target is direct, db_target_path names that database, and portable_source_db is omitted.

This separation means:

• You can gitcrawl embed against a portable store without dirtying the Git checkout
• Local cluster overrides (close-cluster, exclusions, canonicals) live in the runtime mirror
• Before publishing, even refresh writes only to the runtime mirror and leaves the checkout database untouched
• The portable prune publishing step is the boundary that writes a validated database and manifest back into the checkout

Publishing: gitcrawl portable prune

gitcrawl portable prune
gitcrawl portable prune --body-chars 256       # default
gitcrawl portable prune --body-chars 512 --no-vacuum
gitcrawl portable prune --include-sync-failures # opt-in, error text redacted
gitcrawl portable prune --no-publish            # prune only the runtime mirror
gitcrawl portable prune --json

portable prune validates the pruned runtime-mirror database with SQLite quick_check and writes a .manifest.json next to it with size and SHA-256 integrity. When the configured db_path is inside a portable checkout, prune then stages the database and manifest inside that checkout, validates the staged pair, and replaces the published files with atomic renames while preserving their existing permissions. A failed publish restores the previously published pair; an interrupted one is caught later by manifest validation and the standard repair path, the same way an interrupted git pull of the store is. Pass --no-publish to leave the pruned result only in the runtime mirror. Consumers use the manifest to reject incomplete or mismatched portable-store downloads before replacing a known-good runtime mirror.

prune converts the database into the portable v2 backup format and (by default) runs SQLite VACUUM to reclaim space. The result is a smaller database and matching manifest ready to commit from the portable checkout. JSON output includes published, which is true only when this copy-back happened, plus published_db_path and published_manifest_path when published.

Portable v2 keeps the data agents most often need for offline GitHub reads:

• Repositories, issues, pull requests, labels, author login/type/association, and timestamps
• Compact issue/PR body excerpts plus original body lengths
• Compact comments, reviews, and review-comment excerpts plus original body lengths
• PR details, files, commits, status checks, and workflow runs
• Thread revisions, deterministic fingerprints, and key summaries used by duplicate and cluster-oriented workflows

It strips the data that is large, private, easy to regenerate, or mainly useful for exact API replay: raw GitHub JSON, generated documents and FTS indexes, embeddings and vectors, code snapshots and diff blobs, cluster run history, the sync failure ledger, similarity edges, and blob storage. Pass --include-sync-failures only when failure history is useful to portable-store readers; the table is retained but every error_message is replaced with [redacted for portable export]. Once the source schema contains the ledger, pruning securely rewrites the database even with --no-vacuum so current, deleted, and historical retry text cannot remain in free pages. An interrupted rewrite remains marked pending and is retried by the next prune. The database records this contract in portable_metadata with schema=gitcrawl-portable-sync-v2, includes, excluded, capabilities, and thread_author_profile keys. The added revision, fingerprint, summary, and author-association fields are additive; the portable schema identifier remains v2 so older readers can continue using the columns and tables they understand.

Portable mirrors retain existing revision-bound key summaries, but do not regenerate them from compact body excerpts. Pruning removes canonical revision evidence blobs, so run a fully hydrated sync in a writable archive before summarize; the summary queue requires the exact content-addressed payload bound to the revision.

After pruning, commit and push both the database and its .manifest.json from the portable checkout the way you would for any Git repository.

A typical publishing flow

# In the portable store checkout, refresh upstream data into the local runtime mirror.
gitcrawl refresh owner/repo

# Prune for a small, shareable footprint and publish the database plus manifest into the checkout.
gitcrawl portable prune --body-chars 256

# Commit and push using normal Git from the configured portable checkout.
cd ~/Library/Application\ Support/gitcrawl/stores/gitcrawl-store
# Linux default: cd ~/.config/gitcrawl/stores/gitcrawl-store
git add data/openclaw__openclaw.sync.db data/openclaw__openclaw.sync.db.manifest.json
git commit -m "data: refresh openclaw/gitcrawl"
git push

Other agents and machines pull the new commit on their next read-only command.

Cached search against a portable store

gitcrawl search works against portable-store data with one wrinkle: when the portable store has been pruned, generated document indexes may not be present. Search falls back to compact thread title/body data automatically — you keep useful results without the publisher needing to ship the full document indexes.

The v2 backup also keeps comments and PR-detail tables for local review, clustering, search, and TUI workflows.

Caveats

• The portable store carries the SQLite database. It does not carry the Octopool gh cache.
• Vectors regenerated on each consumer's machine after embed are not shared; portable pruning removes vector tables from the published database.
• Portable stores are read-mostly. Multiple writers pushing concurrently — including concurrent portable prune publishers against the same checkout — race the way any Git workflow does; gate writes through a single publisher or a CI workflow.

See also

• Sync — what gets written into the database that ends up in the portable store
• gh shim migration — Octopool owns pooled gh reads now