Deploying Aspen

Every way to run Aspen runs the same binary, aspen serve, configured from the environment (the contract below):

The container image is built from docker/Dockerfile; releases publish it as ghcr.io/reason-machines/aspen:v<version> for linux/amd64 and linux/arm64 (Releases).

Configuration contract

Aspen reads its configuration from the environment. Every deployment path sets the same variables; aspen serve --help lists them with their defaults.

Request rates. ASPEN_LIMITS limits requests per caller (a token's subject, or an anonymous client address) and per instance. Every deploy path in this directory sets it by default to

{"read": {"burst": 100, "per_second": 20}, "push": {"burst": 20, "per_second": 5}}

(compose in compose/limits.json, Helm in limits, the Terraform modules in aspen_limits_json, the AWS module also through container_environment, which overrides it; the public demo passes stricter ones that way: read 60 then 10/s, push 20 then 2/s, and a push quota). A fleet of agents sharing one token shares one budget, so raise the rates for it rather than turning them off. aspen serve without read or push rates logs a warning at startup.

Secret files. ASPEN_SSH_HOST_KEY_FILE and the mirror credentials' *_FILE variables name secrets (in aspen serve, aspen import and aspen mirror), so Aspen opens each one once (following symbolic links) and checks what it opened before reading it: anything but a regular file (a directory, a FIFO, a device) stops startup, as does a file other users may access (any permission for "other"), with an error naming the variable and the problem; and a file a group Aspen does not run in (neither its effective group nor a supplementary one) may access is read with a warning. Use chmod 600, or 640 with Aspen's group (a Kubernetes Secret mounted with defaultMode: 0440 and the pod's fsGroup, as the Helm chart does, passes).

Images. Every image these files pull by default is pinned to a digest: the Dockerfile's base images and syntax frontend, compose's PostgreSQL and MinIO, the Hetzner module's PostgreSQL, MinIO and Caddy, and the AWS placeholder. deploy/scripts/check-image-pins.py (required CI) refuses an unpinned one. Images you pass in (the Aspen image itself) must be pinned to a version tag (:v1.2.3) or a digest in every Terraform module, AWS included.

Set ASPEN_TRUSTED_PROXY_HOPS to exactly the number of proxies that append to X-Forwarded-For. Too low, and every caller has the nearest proxy's address: one caller's bad credentials then lock out every credential, and all anonymous callers share one rate limit. Too high, and Aspen reads an entry the client wrote itself: a client that sends its own X-Forwarded-For chooses the address it is limited by on every request, so its credential guessing is not limited, and nothing shows it. Only callers that send no header fall back to the nearest proxy's address (reason="missing" below). Aspen reads the header only on connections from ASPEN_TRUSTED_PROXIES, parses each entry on its own (an entry may carry a port, and a client's garbage entry cannot hide the proxies' entries after it), and keys a request whose entry it cannot read by its TCP peer. Each such request counts in aspen_client_address_fallbacks_total{reason} (untrusted_peer, missing, malformed); a steady rate means the hop count or the proxy list is wrong, or something reaches Aspen around its proxy. The default list trusts every private, shared and link-local address, so anything that reaches Aspen from one of those without going through the proxy (another pod in the cluster, a host on the VPC or tailnet) can choose its address the same way. Narrow ASPEN_TRUSTED_PROXIES to the addresses your proxy connects from (the ingress controller's pod CIDR, the load balancer's subnets), or allow only the proxy to reach Aspen's HTTP port (a NetworkPolicy, or a security group as the AWS module uses). SSH limits always key on the TCP peer, so a load balancer in front of SSH must preserve client addresses: the AWS module's NLB does (preserve_client_ip), and so does the chart's SSH Service (externalTrafficPolicy: Local). Aspen does not parse proxy protocol. See docs/SECURITY.md, accepted risks.

aspen serve refuses to start, naming the variable, when a setting is unusable: a DATABASE_URL that carries a password, a URL secret without a file:// store, a policy that does not parse. It answers GET /healthz (the process serves HTTP) and GET /readyz (200 once the metadata database answers a query, 503 otherwise); point liveness checks at the first and load balancers at the second. /-/metrics is only on the metrics listener (aspen dev also serves it beside Git), with the job, queue-lag and wider-trunk gauges read from the database at most every 30 s. Git over SSH shares the HTTP server's repositories, access decisions and rate-limit buckets. SIGTERM stops accepting connections and finishes the requests in flight. It speaks HTTP/1.1 and cleartext HTTP/2 (h2c) on the same port.

Process roles

By default every aspen serve process serves Git and also runs the background worker: compact jobs (covers over each ref's versions), index_commits jobs and garbage collection. Under a high push rate the worker competes with pushes for CPU, disk and the database pool on the same instances, and adding Git servers is the only way to add compaction. ASPEN_ROLE splits the two:

Every process reads the same DATABASE_URL and ASPEN_BLOB_URL. Jobs are leased, so any number of all and compact processes share the queue, and garbage collection is safe on all of them at once. A deployment with serve processes must run at least one compact (or all) process, or covers are never built, large pushes' commits are never indexed and garbage is never collected. ASPEN_ROLE=compact refuses ASPEN_SSH_LISTEN and ASPEN_MERGE_QUEUE=on, which only a Git server uses. Scale the compact processes on aspen_compaction_backlog: flat under a steady push rate means compaction keeps up (docs/OBSERVABILITY.md, "Compaction backlog"). The Helm chart (compactor.enabled) and the AWS module (compactor_count) run them as their own Deployment and ECS service.

Repositories and API tokens are created with the same binary, against the same DATABASE_URL:

aspen repo create acme/app                       # trunk refs/heads/main
aspen token create --subject agent:ci-7 --grant write:acme/app   # printed once
aspen token list                                  # never shows a secret
aspen token revoke <id>
aspen key add ~/.ssh/id_ed25519.pub --subject user:alice --grant write:acme/*   # Git over SSH
aspen key list
aspen key revoke SHA256:...

Git sends the token as the password: git push https://agent:<token>@git.example.com/acme/app.git.

The password is kept out of DATABASE_URL so it can come from a secret store on its own; Terraform never sees it in plain text (see below).

CDN URLs are signed by Aspen with ASPEN_EXPRESS_KEYS and verified by aspen-express, the CDN's origin, not by the CDN (decisions log, node C2). The AWS Terraform below follows that design: CloudFront's origin is the servers' own /-/express, and the module generates the signing key and the SSH host key into Secrets Manager.

---

On your machine

cargo install --locked --git https://github.com/reason-machines/aspen aspen-cli   # or a release archive (below)
aspen dev                                # 127.0.0.1:7070, data in ./.aspen
git push http://127.0.0.1:7070/demo.git main
git clone http://127.0.0.1:7070/demo.git

aspen dev is aspen serve with SQLite and local disk under --data-dir, the demo repository (more with --repo <name>), and packfile URIs served from its own /-/blobs route. It lets anyone who can reach it push, so it listens only on a loopback address and refuses any other. Once a Homebrew tap exists (not yet: publishing waits on the release gate, see "Releases"), brew install reason-machines/tap/aspen installs the same binary and brew services start aspen runs it in the background with its data in Homebrew's var/aspen.

---

Single host (docker compose)

You need Docker with the compose plugin (docker compose version).

cd deploy/compose
./init-env.sh            # writes .env with random passwords; never overwrites
docker compose up -d
docker compose ps        # postgres, minio and aspen should be "healthy"
curl http://127.0.0.1:7070/healthz    # -> ok

What runs:

• postgres (postgres:17-alpine, pinned to a digest in compose.yaml), data in the postgres-data volume.
• minio, data in the minio-data volume. Upstream MinIO no longer publishes container images, so the default is Chainguard's maintained build (cgr.dev/chainguard/minio:latest-dev, pinned to a digest in compose.yaml). Set MINIO_IMAGE in .env to use your own.
• minio-init, a one-shot job that creates the aspen-packs bucket and a MinIO user aspen that can read and write objects in that bucket only. It is idempotent and re-applies ASPEN_S3_SECRET_KEY on every up, so rotating the key in .env works.
• aspen, aspen serve built from docker/Dockerfile on the first up (set ASPEN_IMAGE to use a published image), with its push spool on the aspen-spool volume. Its metrics listen on port 9464 inside the compose network only; SSH is off until you add a host key (comment in compose.yaml). Request rates per caller come from compose/limits.json. Its HTTP port binds to 127.0.0.1; set ASPEN_BIND_ADDRESS=0.0.0.0 in .env to expose it. PostgreSQL and MinIO publish no ports, so packfile URIs are off (ASPEN_PACKFILE_URIS): clients could not reach http://minio:9000.

Create a repository and a token, then push:

docker compose exec aspen aspen repo create acme/app
docker compose exec aspen aspen token create --subject me --grant write:acme/app
git push http://me:<token>@127.0.0.1:7070/acme/app.git main

Settings you can put in .env: ASPEN_IMAGE, ASPEN_HTTP_PORT, ASPEN_BIND_ADDRESS, ASPEN_PUBLIC_URL, ASPEN_PACKFILE_URIS, ASPEN_BUCKET, MINIO_ROOT_USER, MINIO_IMAGE, POSTGRES_IMAGE, ASPEN_LOG.

Stop with docker compose down; add --volumes to delete all data.

Smoke test

compose/smoke.sh starts an isolated copy of the stack (own project name, port and throwaway passwords), checks it, and tears it down: every service healthy, /healthz answers, PostgreSQL accepts Aspen's DATABASE_URL and PGPASSWORD over TCP (and rejects a wrong password), Aspen's S3 credentials can write, read and delete objects in the bucket but cannot create buckets, and Aspen serves Git end to end: a repository and token from the admin commands, an anonymous push refused, a token push stored as a pack in MinIO, a clone that passes git fsck --strict, the same clone after Aspen restarts, and a clean exit on SIGTERM. CI runs it on pull requests that touch deploy/ and nightly (.github/workflows/deploy.yml), against the image it just built (ASPEN_IMAGE=aspen:ci deploy/compose/smoke.sh).

The Aspen image

docker build -f deploy/docker/Dockerfile -t aspen .    # from the repository root
docker run --rm aspen --version
docker run --rm aspen serve --help                     # the default command is `serve`

Multi-stage: compiled with the pinned Rust toolchain, shipped on debian:trixie-slim with CA certificates only, running as uid 10001, with ASPEN_LISTEN=0.0.0.0:7070 and ASPEN_SPOOL_DIR=/var/lib/aspen/spool (mount a volume there). There is no curl or wget: health checks run aspen healthcheck [--url http://127.0.0.1:7070/readyz], which exits 0 only on a 200. The Dockerfile's ARG RUST_VERSION must equal channel in rust-toolchain.toml; required CI runs deploy/scripts/check-rust-version.sh to enforce it, so change both together when you bump the toolchain. Its release target packages binaries built elsewhere (the release workflow), so multi-arch images need no emulated compile.

---

AWS (Terraform)

terraform/aws is a reusable module; terraform/aws/examples/isolated is a root module that uses it. terraform/aws/public-demo is how Aspen's own public demo runs on it (an account stack with the budget alarm, DNS zone and image repositories, and staging and demo environments with the place canvas), with its runbook.

Isolation comes first

Deploy Aspen into a dedicated AWS account (an AWS Organizations member account works well). At minimum use a dedicated VPC, which the module always creates; never point it at an account that runs other production systems.

• The example root sets the AWS provider's allowed_account_ids to aws_account_id, so Terraform refuses to run with credentials for any other account.
• The module shares nothing with existing infrastructure except the Route53 hosted zone you pass in, where it adds records for its own hostnames.
• Remote state lives in a bucket in the same dedicated account.

What it creates

Run it

Prerequisites: Terraform 1.11 or newer, credentials for the dedicated account, and a public Route53 hosted zone for a domain you already own.

1. State bucket (once per account):

   aws s3api create-bucket --bucket <state-bucket> --region us-east-1
   aws s3api put-bucket-versioning --bucket <state-bucket> \
     --versioning-configuration Status=Enabled
   aws s3api put-public-access-block --bucket <state-bucket> \
     --public-access-block-configuration \
     BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true
   

   (Outside us-east-1, add --create-bucket-configuration LocationConstraint=<region>.)

2. Configure and apply:

   cd deploy/terraform/aws/examples/isolated
   cp backend.hcl.example backend.hcl                 # state bucket, key, region
   cp terraform.tfvars.example terraform.tfvars       # account, domain, emails
   terraform init -backend-config=backend.hcl
   terraform plan -out=aspen.tfplan
   terraform apply aspen.tfplan
   curl "$(terraform output -raw http_url)/healthz"   # -> ok
   

   backend.hcl, terraform.tfvars and state files are ignored by git. AWS emails each budget recipient once to confirm the subscription.

3. Switch to the real image: copy a release image to ECR (aws ecr create-repository, then skopeo copy --all docker://ghcr.io/reason-machines/aspen:v<version> docker://<ecr-repo>:v<version>), set container_image and ecr_repository_arns, and apply again. Then create a repository and a token with ECS Exec (aws ecs execute-command --cluster <name> --task <task> --container aspen --interactive --command "aspen repo create acme/app"). With SSH on, publish the host key clients pin: the ssh_host_public_key_parameter output names the SSM parameter that holds it (aws ssm get-parameter --name <parameter>).

Tearing down

The database has deletion protection and the pack bucket refuses to delete while it holds objects. To destroy everything, set db_deletion_protection = false (and bucket_force_destroy = true if you want the packs gone), apply, then terraform destroy. RDS keeps a final snapshot named <name>-final; delete it by hand when you no longer need it.

Rotating the database password

Bump db_password_version and apply: a new password is generated, set on the instance and written to Secrets Manager in the same run. Then force a new ECS deployment so tasks pick it up: aws ecs update-service --cluster <name> --service <name> --force-new-deployment.

Cost of the default footprint

Approximate us-east-1 list prices, 730 hours a month, idle (no traffic):

Knobs: nat_gateway = true adds about $33 plus $0.045/GB processed; enable_ssh = false saves about $24; fargate_spot = true cuts the Fargate line by about 70%; db_multi_az = true doubles the RDS line. The default task (0.25 vCPU / 0.5 GiB) suits trying it out: size container_memory for real pushes and lower ASPEN_PREFETCH_MEMORY_BYTES and the push limits to fit it (the public demo's environment root shows one sizing). Traffic costs come on top: CloudFront egress is about $0.085/GB in North America and Europe after the first 1 TB each month, which is why it has its own budget alarm.

Checks without an AWS account

See Checks without a cloud account: the module tests in terraform/aws/tests plan the module against mocked providers and assert the properties above: private, encrypted database; locked-down bucket with multipart cleanup; the $300 budget with a required recipient; the CDN in front of /-/express with a path-only cache key and the viewer function that refuses expired URLs (whose own tests are node terraform/aws/cdn/viewer-request.test.mjs); generated secrets written write-only; no credentials in plain environment variables; and the placeholder/real-image switch.

---

Kubernetes (Helm)

helm/aspen deploys aspen serve (two replicas by default) and, optionally, standalone aspen express cache nodes. It does not run PostgreSQL or object storage: bring a PostgreSQL 14+ database and an S3-compatible bucket (S3, GCS through its S3 API, R2, MinIO).

kubectl create secret generic aspen-db --from-literal=password='<db password>'
helm install aspen deploy/helm/aspen \
  --set database.url='postgres://aspen@db.internal:5432/aspen?sslmode=require' \
  --set database.passwordSecret.name=aspen-db \
  --set blob.url=s3://acme-aspen-packs --set blob.env.AWS_REGION=us-east-1 \
  --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::123456789012:role/aspen \
  --set ingress.enabled=true --set ingress.host=git.example.com --set publicUrl=https://git.example.com
helm test aspen
kubectl exec deploy/aspen -- aspen repo create acme/app

What the chart guarantees (checked by helm/check-chart.py in required CI, and end to end on a kind cluster in the deploy workflow by helm/ci/kind-e2e.sh):

• helm install fails, naming the value, without database.url, with a password in it, with SQLite (replicas share one database), without database.passwordSecret, with a file:// store, or with express settings but no express.keysSecret;
• secrets reach pods only through secretKeyRef; a secret-looking name (*SECRET*, *TOKEN*, *PASSWORD*, *_KEY) with a plain value in blob.env or extraEnv fails the install; the policy and limits are a ConfigMap whose change rolls the pods;
• mirror mode (docs/MIRROR.md): mirror.credentials names each credential with a tokenSecret or a githubApp whose private key is a Secret, and the chart sets the ASPEN_MIRROR_* variables from them, secrets through secretKeyRef only;
• pods run as uid 10001 with a read-only root file system, no capabilities and no service-account token;
• the push spool is node disk (emptyDir with a size limit) or a per-pod PersistentVolumeClaim (spool.ephemeralVolume), never memory;
• liveness probes /healthz, readiness /readyz; a pod gets 300 s to finish long clones and pushes when stopped, and a PodDisruptionBudget keeps one replica up;
• /-/metrics listens on its own container port (9464, advertised with prometheus.io/* pod annotations), which neither the Service nor the ingress exposes; with metrics.networkPolicy.enabled, a NetworkPolicy keeps Git open to every source and admits only the peers in metrics.networkPolicy.from (your Prometheus) to that port, and install fails without any (it needs a network plugin that enforces NetworkPolicy).