Security model and review
This document describes what Aspen defends against, how each public surface is protected, and the results of the pre-launch security review (node D4 in PLAN.md). Read it before you expose an Aspen instance to the internet.
Reporting a vulnerability. The repository is private and no disclosure address has been chosen yet. Until one is published here, report issues privately to the maintainers; do not open a public issue.
1. Threat model
Assets
| Asset | Where it lives | Why it matters |
|---|---|---|
| Repository contents | Packs in the object store, metadata in Postgres/SQLite | Private code; integrity of every clone |
| Ref state | refs / ref_versions tables |
Who can move a branch decides what ships |
| Credentials | API tokens (SHA-256 digests only), SSH keys (public keys only), OIDC trust configuration | A stolen credential is an identity |
| Server secrets | SSH host key, /-/blobs URL key, express signing keys, mirror tokens and GitHub App keys, PGPASSWORD |
Impersonating the server, forging download URLs, writing upstream |
| Availability and budget | CPU, memory, disk, database connections, egress | The public demo has a $300/month budget |
Actors
- Anonymous internet clients. They can reach the Git HTTP listener,
the SSH listener,
/healthzand/readyz, signed blob and express URLs they are given, andplace's API. They may hold public SSH keys of real users (these are often published) and may control many addresses (an IPv6 /48 gives 65,536 /64s; botnets). - Authenticated callers with a token, SSH key or OIDC identity, with grants on some repositories, who try to reach other repositories, refs outside their write patterns, or to exhaust shared resources.
- Pushers control the bytes of every pack they send (zip bombs, delta cycles, deep chains, duplicate objects, malformed objects).
- A compromised upstream in mirror mode, which controls what Aspen fetches.
- Out of scope: an attacker with access to the database, the object store, the host, or the operator's secret store; side channels on shared hardware.
Security goals
- No request reaches repository data without an access decision for that repository, and a hidden repository is indistinguishable from a missing one.
- A caller can write only refs its grants allow, and only through the push path's checks.
- Secrets never reach logs, error messages, URLs, process listings or Terraform state.
- Every input a stranger controls is bounded in bytes, count, memory and time before the server commits resources to it.
- Stored packs keep their invariants (self-contained, base-first, no duplicates) whatever a pusher sends.
2. Surfaces and their controls
Authentication (crates/aspen-auth)
- API tokens are
aspen_<id>_<secret>with a 256-bit secret; only its SHA-256 is stored and digests are compared in constant time (TokenDigesthas noPartialEq). A presented but bad credential is 401, never downgraded to anonymous. - OIDC identity tokens are verified with
ring: RS256/384/512 and ES256/384 only (noneand HMAC refused), key type andalgmust fit,critrefused, theniss,aud,exp,nbf(numbers only),sub. A token is verified only by the issuers whoseissit names, so a forged token costs one issuer's signature checks (D4). - SSH keys: Ed25519 and ECDSA only (RSA refused, no
rsacrate), looked up by fingerprint after russh verified the client's signature. The fingerprint is public, so only the SSH transport builds an SSH-key credential; no HTTP header ever becomes one. - Failed authentications are limited per client address (D4):
LimitsConfig::auth_failures, on by default (100 at once, then one a second). Every presented credential that fails, over HTTP, over SSH (a key offer, and each re-check of an authenticated key before a command, issue #243) and atplace's kill switch, drains the address's bucket; while it is empty, a request that presents a credential is refused (HTTP 429, SSH key refused) before the credential costs a database read or a signature check. Each credential takes its token before it is checked and gives it back when it authenticates (or the store is down), so requests that arrive together cannot all pass on one token: however many run at once, at most the bucket's level of credentials are checked (issue #198).
Authorization (aspen_auth::policy, aspen_server::access)
- One decision per request, by repository name, before the repository is
looked up: anonymous callers without a grant get 401, signed-in callers
without a grant 404 (the same as a missing repository), read-only
callers that push 403. Every push command is checked again against the
caller's ref patterns (
ng <ref> permission denied). - One route parser (
GitRoute) for HTTP and SSH; handlers take the repository only from theAuthorizedextension the guard inserts. SeveralAuthorizationheaders are refused (400). - Tokens and keys hold exactly their own grants plus the anonymous ones.
Git over HTTP (aspen_server, aspen serve)
- Listener (
aspen_server::listen, D4): request headers must arrive within 30 s of the first byte (axum::servesets no hyper timer, so before D4 headers never timed out); HTTP/2 connections are pinged every 30 s and carry at most 100 streams. - Bodies: fetch bodies are read whole, at most 9 MiB as sent and after
gunzip (#118; the v2 parser takes 8 MiB); push bodies stream under the
per-push byte cap and the caller's hourly byte budget. Every Git request
body fails after 60 s with no bytes (
ServerConfig::body_idle_timeout, D4), and after its first 10 s of waiting once it averages under 1 KiB/s (ServerConfig::body_min_bytes_per_sec, #118). - Concurrent work (
aspen_server::admission, #118): fetches, pushes and stitches each hold a slot from admission (before the body is read) until they end, however they end. Per caller (the rate-limit key, over HTTP and SSH together;LimitsConfig::concurrent, 16/4/8 by default) an extra request is429; per instance (ASPEN_MAX_CONCURRENT_*, 32/16/32) it waits in a bounded queue (64, at most 10 s) and is then503; both withRetry-After, and a retryableERRover SSH and for a stitch. - Protocol limits (A1, A1b): byte, line and count caps on every pkt-line request, negotiation rounds, ref names, push options.
- Pack ingest (A2, B1): object count checked against the header before the body is read; every declared object and delta size checked against the object limit and the inflated total before gix allocates; delta depth capped at 4095; duplicate objects refused; prefetch memory shared across pushes through a gate.
- Rate limits (B3): per-caller token buckets for reads and pushes; anonymous callers are keyed by IPv4 address or IPv6 /64. When a limiter's table is full, only refilled buckets are dropped; otherwise the new caller is refused until one refills, so nobody can reset a drained bucket by churning addresses (D4).
- Health:
/healthztouches nothing;/readyzanswers from a database check at most one second old, shared by concurrent probes (D4). - Metrics are on a private listener (
ASPEN_METRICS_LISTEN, default127.0.0.1:9464), never the Git listener except in loopback-onlyaspen dev.
Git over SSH (aspen_server::ssh, SSH.md)
- Public-key authentication only; session channels only; one Git command
per session with Git's own quoting parsed without a shell; only
GIT_PROTOCOLfrom the environment. The key is identified again for every command (and before every protocol v2 command after the first), through the same charged check as an offer: a failed re-check drains the address's failed-authentication bucket, and while it is empty the command is refused without a lookup (issue #243). - Admission (D4): at most 4,096 connections in total and 10 unauthenticated connections per address; a connection that has not authenticated within 30 s is disconnected, key exchange included.
- Offers (D4): russh does not count an accepted unsigned key offer as an attempt, so a client holding only a registered public key could make one database read per repeated offer. Each offered fingerprint is now looked up once per connection, at most 20 distinct keys per connection, and refused keys drain the address's failed-authentication bucket.
- Commands refused before they reach a repository, and shell greetings, cost a request from the subject's bucket (D4).
Signed URLs (/-/blobs, aspen-express, presigned bucket URLs)
- HMAC-SHA256 over the key and expiry (domain-separated for express), with
the signature and expiry in the path for express so a CDN never caches a
forged URL. Every refusal is the same
403. TTLs are 15 minutes by default and at most 1 hour (RECOMMENDED_MAX_PRESIGN_TTL). /-/blobsresponses carryX-Content-Type-Options: nosniffandContent-Security-Policy: sandbox(D4), since a pusher chose the bytes.
Mirror mode (aspen-mirror)
- Credentials are named, read from the environment or files, never stored in the database or logged; sent only over HTTPS or to loopback; redirects followed only within the same origin. Imported history goes through the push path, so every stored-pack invariant holds for it.
Secrets and configuration (aspen-cli, deploy/)
DATABASE_URLmust not carry a password (PGPASSWORDinstead), checked byaspen serve, the admin commands,aspen mirrorandaspen doctor jobs(the last two since D4), and hidden in--helpoutput.- Types holding secrets redact themselves in
Debug(Secret,TokenDigest,UrlKey,LocalUrlSigner, SSH settings, mirror credentials, and since D4 the/-/blobsURL key inServeConfig). - The SSH host key is never generated; it comes from a file or the secret store and is never echoed.
- Terraform keeps the database password out of state (write-only
attributes), CloudFront fronts only packs, and tasks accept traffic only
from their load balancers. Behind the AWS ALB,
ASPEN_TRUSTED_PROXY_HOPSis 1, so anonymous limits key on the client, not the ALB (D4). The header is read only on connections fromASPEN_TRUSTED_PROXIES(default: private, shared and link-local ranges, where the ALB connects from), so a client that reaches a task some other way is keyed by its own address (#121).
Dependencies
cargo deny(required CI) checks RustSec advisories, licenses, bans and sources. No advisory is ignored (deny.tomlignore = []).- OpenSSL and aws-lc are kept out (rustls on
ring); thersacrate (RUSTSEC-2023-0071) is kept out by refusing RSA SSH keys and not usingjsonwebtoken.
3. Review results (D4)
Three independent read-only audits (HTTP surface; SSH and the auth crate; secrets, deployment and dependencies) plus the obligations earlier nodes recorded for D4 in the PLAN decisions log.
Fixed in the D4 pull request, each with a test
| # | Finding | Severity | Test (mutation in crates/aspen-server/scripts/check-mutations-d4.py) |
|---|---|---|---|
| 1 | Failed authentication was not rate limited: each bad token cost a database read, each bad JWT a signature check per issuer and key, over HTTP, SSH and place's kill switch (B3 obligation 1, C7 obligation 1) |
High | failed_authentications_are_limited_per_address in aspen-server/tests/access.rs and ssh_limits.rs, auth_failures_are_limited_per_address_and_successes_are_free (D2, D3, D7) |
| 2 | A JWT was verified by every configured issuer | Medium | a_forged_token_is_checked_by_the_issuer_it_names_only (D4) |
| 3 | A repeated unsigned SSH offer of a registered public key cost a database read each time, unbounded per connection | High | repeated_offers_of_a_registered_key_cost_one_lookup, distinct_offers_per_connection_are_capped (D5, D6) |
| 4 | SSH had no connection caps, no per-address limit and no login deadline (C3 obligation) | High | connections_are_capped_in_total, unauthenticated_connections_per_address_are_capped, a_connection_must_authenticate_within_the_deadline (D8 to D10) |
| 5 | A full limiter table dropped the least recently used half, resetting drained buckets (B3 obligation 2, C7 obligation 4) | Medium | a_full_table_refuses_new_keys_and_resets_nothing (D1) |
| 6 | HTTP request headers never timed out (no hyper timer under axum::serve) |
Medium | slow_headers_time_out_and_requests_see_their_peer (D12) |
| 7 | A stalled request body held its request (and for a push a blocking thread) forever (B2 follow-up) | Medium | an_idle_body_times_out_and_a_slow_one_does_not, a_stalled_git_request_body_is_cut_off (D13, D14) |
| 8 | /readyz ran a database query per anonymous request |
Medium | readyz_follows_the_metadata_database (D15) |
| 9 | The AWS module did not set ASPEN_TRUSTED_PROXY_HOPS, so all anonymous callers behind the ALB shared one bucket |
High (deployment) | deploy/terraform/aws/tests/module.tftest.hcl assert |
| 10 | aspen mirror and aspen doctor jobs accepted a password in DATABASE_URL; aspen mirror --help showed its value |
Low | operator_commands_refuse_and_hide_a_password_in_the_database_url (D16 to D18) |
| 11 | ServeConfig's derived Debug would print the /-/blobs URL key |
Low | the_local_url_secret_needs_a_file_store_a_public_url_and_32_bytes (D19) |
| 12 | SSH commands refused before reaching a repository, and shell greetings, cost a key lookup but no request | Low | refused_commands_and_greetings_draw_on_the_request_bucket (D11) |
| 13 | /-/blobs responses lacked nosniff |
Low | the_blob_route_checks_signatures_and_serves_ranges (D20) |
| 14 | (Found in review of this PR.) With the failure table full of drained buckets, a new address was admitted with a full bucket and its failures could not be charged, so the failure limit stopped working; a push budget admitted a new caller the same way | High | a_full_table_refuses_unchecked_credentials_from_new_addresses (D21) |
| 15 | (Found in review of this PR.) Requests without a client address shared one failure bucket, so a few bad credentials could lock out every credential; place built its kill-switch key by hand and answered a limited caller 403 |
Medium | an_unknown_address_does_not_share_a_failure_bucket, bad_operator_credentials_drain_the_git_failure_bucket (D22 to D24) |
| 16 | (Found in review of this PR.) IPv4 clients of a [::] listener are IPv4-mapped IPv6 addresses; keyed by /64 they would all share one bucket |
Medium | anonymous_ipv6_callers_are_keyed_by_their_slash_64 (D25) |
Filed as issues (label security)
| Issue | Finding | Severity |
|---|---|---|
| #118 | No per-caller or per-instance cap on concurrent fetches, pushes and stitch work; ls-refs in memory; push bodies on blocking threads. Fixed: fetches, pushes and stitches are capped per caller (429) and per instance (bounded queue, then 503), both with Retry-After; fetch bodies at most 9 MiB; bodies under 1 KiB/s after 10 s fail. ls-refs still builds its response in memory, now under a fetch slot |
Medium-high |
| #119 | ref-prefix lines multiply database work and can exceed Postgres's bind-parameter limit. Fixed: at most 1,024 lines per request; the store drops repeated and covered prefixes, skips ranges behind the page cursor and binds at most 250 ranges per statement (SQLite already failed at about 1,000) |
Medium |
| #120 | AWS NLB and Helm SSH Service hide client addresses, so per-address SSH limits would key on the load balancer (D5) | Medium |
| #121 | X-Forwarded-For trusted from any peer; ports and short headers fall back to one shared bucket. Fixed: the header is read only from ASPEN_TRUSTED_PROXIES peers, entry by entry from raw bytes, ports stripped; an unreadable entry keys by the TCP peer and is counted (aspen_client_address_fallbacks_total) |
Low-medium |
| #122 | Release and CI workflows use tag-pinned actions and unverified tool downloads | Medium (before publishing) |
| #123 | Express cache fills have no concurrency or disk cap. Fixed: fills are capped and reserve their size against the cache budget before copying (docs/EXPRESS.md, limits) |
Low |
| #124 | place: SSE streams uncapped and unbounded in age; start_swarm checks in two steps (C7 obligations 2, 3). Fixed: streams are capped per address (16) and in all (4,096) and end within 10 minutes; a start checks and charges in one step (demo/place/server/README.md) |
Low-medium |
| #125 | Deployment hardening: no default request rates, images not digest-pinned, secret file modes, metrics NetworkPolicy. Fixed: every deploy path sets read and push rates and aspen serve warns without them; default images are digest-pinned (deploy/scripts/check-image-pins.py) and the AWS module refuses an unpinned image; secret files other users can read stop startup; the Helm chart has an optional metrics NetworkPolicy (deploy/README.md) |
Low |
| #198 | The failed-authentication limit overshoots under concurrency: requests from one address that arrive together all pass the check before any failure is charged (raised on #121). Fixed: a credential takes its token before it is checked and gives it back when it authenticates or the store is down (Limits::charge_credential); concurrent_bad_credentials_cannot_overshoot_the_failure_bucket (mutations D43 to D46) |
Low |
| #243 | SSH re-checks of an authenticated key (before each command, and before each protocol v2 command after the first) called the key store directly and were never charged to the failed-authentication bucket, so a connection kept open after its key was revoked could make unbounded uncharged lookups. Fixed: every re-check goes through Access::identify with the connection's address; failed_rechecks_of_a_revoked_key_drain_the_failure_bucket, failed_v2_rechecks_drain_the_failure_bucket (mutations D47 to D49) |
Low |
Accepted risks
- Signed URLs outlive revocation by at most their TTL (15 minutes by default, 1 hour at most), the same as presigned bucket URLs (C2 row). Packfile-URI downloads carry no credential, so per-download checks are not possible without a revocation epoch.
- Limits are per instance. With N instances a caller gets up to N times the configured rates (B3 row).
- The failed-authentication limit is per address, so callers behind one NAT share it: an attacker failing from that address also blocks valid credentials from it until the bucket refills (one a second by default). Anonymous requests are never affected. Because a credential holds a token while it is checked (#198), no more credentials from one address are checked at once than its bucket holds (100 by default): callers behind one NAT that present more than that in the same instant get 429 for the rest, until checks finish (milliseconds, unless the database is slow).
- A full limiter table refuses new callers until a tracked bucket
refills, rather than reset spent buckets: a request from a new caller
gets 429, and so does a credential from a new address, before it is
checked. An attacker who drains 100,000 failure buckets (the default
max_keys; two IPv6 /48s hold that many /64s) keeps every new address from signing in for as long as it keeps them drained, about 100 failures from each address every 100 seconds. Addresses already tracked are unaffected, and so are anonymous requests unless request rates are set. - Misconfigured proxy hops either lock out every credential or lift
the failure limit. The failure limit keys on the client address
ASPEN_TRUSTED_PROXY_HOPSpicks out. Set lower than the proxies in front, every caller gets the nearest proxy's address, so one caller's bad credentials lock out everyone's. Set higher, Aspen reads an entry the client wrote: a client that sends its ownX-Forwarded-Forchooses a fresh failure bucket per request, so its guessing is unlimited and shows on no metric. Only callers that send no header fall back to the nearest proxy's address, counted inaspen_client_address_fallbacks_total{reason="missing"}. No header content reaches the unknown address (#121), which only an embedding that serves without a peer address produces. The SSH per-address limits always key on the TCP peer; the AWS NLB (preserve_client_ip) and the Helm Service (externalTrafficPolicy: Local) pass the client's own address through (#120). A load balancer that does not hides every client behind its address. - The default trusted-proxy list trusts every private address. With
hops above 0 and
ASPEN_TRUSTED_PROXIESunset,X-Forwarded-Foris read from any loopback, private, shared (100.64.0.0/10, which tailnets use) or link-local peer. Anything that reaches Aspen from one of those without going through the proxy (another pod in the cluster when the Helm chart's ingress is on, a host on the VPC or a tailnet) can write a made-up header and choose its rate-limit and failure bucket. Accepted as a default because the proxy's addresses differ per cluster; the Helm values and the deploy README say to narrow the list to the ingress controller's or load balancer's addresses, or to let only the proxy reach the HTTP port. The AWS module allows only the ALB's security group to reach it. /-/metricson the Git listener is possible only withmetrics_on_git_listener, which only loopback-onlyaspen devsets.
Not verified
- The deployed public endpoint: nothing is deployed yet (D5). The launch gate item "public endpoint security review done; quotas and kill switch tested" needs D5's deployment checked against this document.
- The AWS Terraform test runs only in CI; its provider plugins do not run in the review's sandbox.