Run two or more copies of Hardcount (high availability)

Run two or more copies of the Hardcount API behind a load balancer. While one copy restarts or crashes, the others keep serving.

Downtime during a restart is fine for you? Use Production on one VM.

What you need

• Two or more hardcount-api copies behind a load balancer.
• A managed Postgres with automatic failover and point-in-time restore.
• A managed Redis. Hardcount uses it only for rate limiting, and all copies share its counts. If Redis goes down after start-up, see Redis going down after boot.
• A connection pooler is optional: Postgres requirements → Connection poolers.

The settings for each copy are in Self-host. Every copy uses the same settings and the same database.

How the copies share the work

Each copy answers API requests. Background jobs are different: billing runs, payment retries (dunning), and sending webhooks and emails must run once, not once per copy. Two things make sure of that, so copies do not bill or send twice:

• One copy runs each job at a time. Before each run of a job, a copy takes a short lease: a row in Postgres that says “this copy runs this job now” (see leader lease). The other copies stand by. If the copy holding a lease dies, another one takes over within about 15 seconds.
• Queued rows are claimed once. Webhooks and emails wait in queue tables (outboxes) until a background worker sends them. A worker skips any row another worker already holds, instead of waiting for it (Postgres SKIP LOCKED).

A few features still assume one copy: the dashboard’s live webhook stream, test-clock advances, and one billing log note. None of them touch money.

Set up the load balancer

1. Point the load balancer’s health check at GET /health/ready.

   curl -fsS https://your-hardcount-host/health/ready

   You see: {"checks":{"api":"ok","database":"ok","scheduler":"..."},"status":"ok"}, with status 200.

2. Set TRUST_PROXY on every copy to the address range (CIDR) of the load balancer. Separate several ranges with commas.

   Why it matters: Self-host → Settings. Set TRUST_PROXY=none only when the copies take client connections directly, with no proxy in front.

   You see: the copy starts. In production, a copy with TRUST_PROXY unset refuses to start with TRUST_PROXY is required in production.

3. Set SHUTDOWN_DRAIN_DELAY to at least the load balancer’s health-check interval times its unhealthy threshold. For an AWS ALB with its defaults, that is 30s × 2, so set SHUTDOWN_DRAIN_DELAY=60s. Use 0 when the orchestrator removes the copy from the load balancer itself, as Kubernetes does. The default is 5s outside local.

   You see: when you stop a copy, /health/ready on that copy answers 503 {"status":"draining"} for that long, while it still serves requests.

4. Set the shutdown grace period (Kubernetes terminationGracePeriodSeconds) above SHUTDOWN_DRAIN_DELAY plus 90 seconds: 120s with the 5s default, 180s with SHUTDOWN_DRAIN_DELAY=60s. The Compose stack sets stop_grace_period: 120s.

5. Check the database connections. Each copy opens up to DB_MAX_OPEN_CONNS connections (default 20). Copies × DB_MAX_OPEN_CONNS must stay below your Postgres max_connections.

Health checks

Endpoint	What it tells you
GET /health	Liveness: 200 while the process runs.
GET /health/ready	Readiness: 200 when the database is reachable. 503 when it is not, or when the copy is shutting down. The body also reports the billing scheduler, which does not change the status code.

Neither endpoint is rate limited or written to the audit log.

What happens when a copy stops

On a stop signal (SIGTERM), a copy first tells the load balancer it is leaving: /health/ready answers 503 while the copy still accepts connections. Then it finishes its work in three stages, each with a time limit.

Step	What happens	Time limit
Readiness drain	/health/ready answers 503 {"status":"draining"}; /health stays 200. The copy keeps listening, so the load balancer has time to stop sending it requests.	SHUTDOWN_DRAIN_DELAY (default 5s outside local)
1. Requests in progress	The dashboard’s live webhook stream is closed first, so it cannot hold this stage open.	Up to 30s
2. A test-clock advance in progress	Stopped at the limit. The clock stays advancing until a copy restarts. See Test-clock advance hung.	Up to 30s
3. Background jobs		Up to 30s

The worst case is SHUTDOWN_DRAIN_DELAY plus 90 seconds. Usually the three stages take under 2 seconds after the drain delay.

A grace period shorter than that can kill a copy (SIGKILL) halfway through. Hardcount is built to survive that: queued webhooks and emails a copy had claimed are picked up again when its lease runs out, and the next copy to start recovers a killed test-clock advance. The cost is that a few webhooks or emails can be sent twice.

Upgrades

Read the upgrade notes in the changelog first. Then take a backup, or note the time if your Postgres has point-in-time restore: going back after an upgrade means restoring the database to that point (Going back). Whether you can upgrade one copy at a time (a rolling upgrade) depends on the release’s migrations, below.

Database upgrades (migrations): RUN_MIGRATIONS_ON_BOOT=true is safe when several copies start at once: they take turns through a Postgres lock, and each one applies only what is still missing. A waiting copy gives up after 30 minutes and is restarted. What is not covered is old code running on a new schema, so the release notes decide the order:

• A release whose notes say it has migrations: stop every old copy, run hardcount migrate once (or start one new copy with RUN_MIGRATIONS_ON_BOOT=true), then start the other new copies.
• A release with no migrations: upgrade one copy at a time.
• Hardcount was called Velox before v0.9.0. v0.9.0 installs fresh only: it refuses a database made by Velox, a 0.8.x release or a build of main (why).

Run hardcount migrate status with the new release first to see the plan. Never roll a migration back in production; see Self-host.

Next: A background job is stuck: what to check when billing or another background job stops.