Operations Runbook

Day-2 operations for a self-hosted Eparch deployment: backup/restore, upgrades, migrations, logs, and troubleshooting. Commands assume the ten-container topology (deploy/docker-compose.prod.yml); substitute docker-compose.monolith.yml where you run the monolith.

Data stores at a glance

Databases and roles: the superuser is eparch; runtime services connect as the non-superuser eparch_app (RLS enforced). The database is eparch in the compose deployments and eparch_dev in local dev.

Backup & restore: PostgreSQL

Backup

Take a compressed custom-format dump as the superuser:

docker compose -f deploy/docker-compose.prod.yml exec postgres \
  pg_dump -U eparch -d eparch -Fc -f /tmp/eparch.dump

docker compose -f deploy/docker-compose.prod.yml cp \
  postgres:/tmp/eparch.dump "./eparch-$(date +%Y%m%d-%H%M%S).dump"

Or straight from the host against the published port:

pg_dump "postgres://eparch:${POSTGRES_PASSWORD}@localhost:5432/eparch" \
  -Fc -f "eparch-$(date +%Y%m%d-%H%M%S).dump"

Schedule this (cron/systemd timer) and ship the dump off-host. The dump includes the encrypted secret store — protect it like a secret, and keep EPARCH_SECRET_KEY safe separately (without it the encrypted secrets are unusable).

Restore

Restore into a running Postgres whose roles already exist (the infra/compose/postgres-init scripts create eparch_app/eparch_readonly on first boot of a fresh volume):

docker compose -f deploy/docker-compose.prod.yml cp \
  ./eparch-20260718-120000.dump postgres:/tmp/restore.dump

docker compose -f deploy/docker-compose.prod.yml exec postgres \
  pg_restore -U eparch -d eparch --clean --if-exists /tmp/restore.dump

Restoring into a brand-new volume: bring postgres up alone first so the init scripts create the roles, then restore, then start the app services.

Backup & restore: MinIO / object storage

Use the MinIO client mc. Point an alias at the endpoint (defaults: minioadmin/minioadmin, bucket eparch-rag):

mc alias set eparch http://localhost:9000 "$MINIO_ACCESS_KEY" "$MINIO_SECRET_KEY"

# Backup
mc mirror --overwrite eparch/eparch-rag ./backup/eparch-rag

# Restore
mc mirror --overwrite ./backup/eparch-rag eparch/eparch-rag

For S3-backed deployments, use your provider’s native snapshot/replication instead. Object storage and Postgres should be backed up together so RAG document references stay consistent.

Upgrade procedure

Migrations are idempotent and the migrate job runs before app services, so the upgrade is roll-forward:

# 1. Back up Postgres (and MinIO) first — see above.

# 2. Pull the new images / rebuild.
docker compose -f deploy/docker-compose.prod.yml --env-file deploy/.env pull
#   (for locally built images: add --build to the up command below)

# 3. Bring the stack up. The migrate init container runs all migrations as the
#    superuser and exits; app services wait for it to complete successfully,
#    then restart with the new images.
docker compose -f deploy/docker-compose.prod.yml --env-file deploy/.env up -d --build

Verify health afterward:

docker compose -f deploy/docker-compose.prod.yml ps      # migrate = Exited (0); apps running
curl -f http://localhost:8031/healthz
curl -f http://localhost:8031/readyz

Roll back by redeploying the previous image tags. Note that a migration that changed the schema is not automatically reversed on rollback — restore from backup if a migration must be undone.

Migrations

Each service owns its migrations under services/<svc>/migrations/ and records its version in a dedicated <svc>_schema_migrations table, so runs are idempotent and independent.

In a compose deployment — re-run the one-shot job:

docker compose -f deploy/docker-compose.prod.yml run --rm migrate

Locally (via the Makefile):

make migrate-up                       # apply all services' migrations
make migrate-down                     # prompts for a service, rolls back ONE step
make migrate-create svc=rag name=add_index   # scaffold a new migration

make migrate-up/migrate-down use POSTGRES_URL if set, otherwise default to postgres://eparch:eparch@localhost:5432/eparch_dev. migrate-down reverses a single step for one service — there is no bulk down; use it deliberately.

Logs

• Compose deployments: docker compose -f deploy/docker-compose.prod.yml logs -f <service> (e.g. gateway, identity, migrate). All services log structured JSON to stdout.
• Local dev (make dev-local): per-service logs are written to logs/dev/<service>.log (and streamed to the terminal, prefixed with the service name), e.g. logs/dev/gateway.log, logs/dev/platform-ui.log.

Traces go to the OTLP endpoint (OTEL_EXPORTER_OTLP_ENDPOINT); in local dev the infra stack ships them to Jaeger (http://localhost:16686).

Troubleshooting

A service won’t start / crash-loops

Most services require POSTGRES_URL, JWT_ISSUER, and JWKS_URL; a missing value fails config.Load() immediately. Check the logs for the exact ... is required message.

Database unreachable

Symptom: services exit at startup or fail queries.

• Confirm Postgres is healthy: docker compose ... ps (should be healthy), or pg_isready -h localhost -U eparch -d eparch.
• Verify POSTGRES_URL points at the right host/DB and the credentials match POSTGRES_PASSWORD / EPARCH_APP_PASSWORD.
• Permission-denied on queries usually means the eparch_app role or RLS policies are missing — confirm the postgres-init scripts ran on the volume.

OPA down

Every service authorizes requests through OPA (OPA_ADDR). If OPA is unreachable, authorization fails closed and requests are denied.

• Check the opa container is running and serving on :8181.
• curl -s http://localhost:8181/health should return {} / 200.
• Verify OPA_ADDR resolves from inside the app containers (http://opa:8181 in compose).

Gateway refuses to start

The gateway is deliberately fail-fast. Common causes:

• Redis unreachable — the gateway will not run without rate-limit/budget controls and exits with Redis unreachable at <addr> — refusing to start. Fix Redis, then it starts. (See below.)
• Missing embedding URL — when ENV != local, an empty LLM_EMBEDDING_URL fails config load (so RAG never silently returns stub vectors). Point it at your OpenAI-compatible embeddings endpoint.
• Missing POSTGRES_URL / JWT_ISSUER / JWKS_URL — all required.

Redis down → fail-closed rate limits

Rate limiting and budget enforcement are Redis-backed. The gateway refuses to start when Redis is unreachable rather than silently disabling those controls. Once running, restore Redis and restart the gateway.

• Check the redis container is healthy; redis-cli -h localhost ping → PONG.
• Confirm REDIS_ADDR (and REDIS_PASSWORD if set) are correct.

Migrate job failed → app services never start

App services depend_on the migrate job completing successfully. If migrate shows a non-zero exit, inspect its logs (docker compose ... logs migrate), fix the cause (usually DB connectivity or a bad migration), and re-run docker compose ... run --rm migrate.

Login fails

Login is brokered to WorkOS AuthKit. Verify WORKOS_API_KEY, WORKOS_CLIENT_ID, and that WORKOS_REDIRECT_URI exactly matches the callback registered in the WorkOS dashboard. Confirm identity is signing tokens (JWT_RSA_PRIVATE_KEY set) and its JWKS is reachable at http://localhost:8011/.well-known/jwks.json.