Operations runbook

What to do when Hardcount is in trouble. This runbook is for whoever runs a Hardcount deployment, often during an incident. Find what you see in the table below and open its page. Each page follows the same order: Symptom, Why, Check (commands), Fix, and Verify.

It covers failures specific to Hardcount. General Postgres, Kubernetes and Stripe troubleshooting is out of scope.

Before you start

Most Check steps call the Hardcount API or the database. Set these once:

export API=https://billing.example.com   # your Hardcount address
export KEY=hc_secret_live_...           # a secret API key for the tenant

SQL steps run in psql as the database owner. Under the app’s row-level-security role you see only one tenant’s rows, or none.

Words on these pages

• Tenant: one account’s data. Many tenants share one database.
• Replica: one running copy of the Hardcount server.
• Tick: one run of a background job, such as one billing pass.
• Lease: a short booking, in the database, that lets one replica run one tick of a job. Like a meeting-room booking, it expires unless it is renewed.
• Fence token: a number on the lease that goes up each time a replica takes it. Work checks the number, so a replica whose lease ran out cannot claim anything.
• Outbox: a table of emails or webhooks waiting to be sent.
• Parked: set aside until something changes (invoices, subscriptions).
• Wait ladder: the growing waits between billing retries (Billing failing).
• PI: a Stripe PaymentIntent, the Stripe object for one card charge.

What do you see?

If you see	Open
The scheduler check in /health/ready starts with degraded, no invoices are being generated, or a background job looks stuck	A background job is stuck
hardcount_email_outbox_failing_emails or hardcount_email_outbox_oldest_due_age_seconds firing, emails dead-lettered, or customers report missing invoice emails	Emails not going out
hardcount_webhook_outbox_oldest_pending_age_seconds firing, or webhook_outbox rows dead-lettered	Webhook outbox backed up
One customer’s webhook endpoint keeps failing (hardcount_webhook_deliveries_total{status="failed"})	A webhook endpoint keeps failing
hardcount_stripe_breaker_state == 2, or dunning retries skipped	Stripe circuit breaker open
Invoices stuck with an unconfirmed payment for hours	A charge’s result is unclear
A test clock stuck on the Advancing badge	Test-clock advance hung
A tenant sees another tenant’s data	Data leak between tenants suspected
LiteLLM spend 503s, or Hardcount spend short of LiteLLM’s	LiteLLM spend gap after an outage
A replica will not boot and names Redis	A replica will not start
Someone cannot sign in: “Too many sign-in attempts”, or hardcount_auth_login_total{outcome="throttled"} rising	Sign-in says “Too many sign-in attempts”
hardcount migrate or the start-up stops with migration … failed, goose_db_version, no migration record or another Hardcount process is migrating	A database upgrade stopped
hardcount migrate or the start-up stops with this database was made by Velox	The database was made by Velox
An upgrade stops with waited 5s for a table lock	A database upgrade stopped: table lock
hardcount_stripe_key_unusable_total rising, or “Stripe refused this tenant’s API key” in the logs: charges paused until the key is fixed in Settings → Payments	Health checks, alerts and logs
hardcount_parked_invoices{disposition="open"} above zero	A charge’s result is unclear: parked
An invoice notice reads “Customer must confirm payment”, or a payment retry was not attempted because the bank wants the customer to confirm	The customer’s bank wants them to confirm a payment
An invoice notice reads “Below Stripe’s minimum charge”, or a customer’s hosted page has no Pay button for a small amount	An invoice is below Stripe’s minimum charge
“stale charge check: can’t confirm an earlier charge with Stripe” in the logs, or hardcount_stale_charge_hold_resolutions_total{result="cannot_ask"} rising	A charge’s result is unclear: interrupted
An invoice says “after a lost dispute”, a payment.dispute_opened or payment.dispute_closed webhook, or Stripe emails about a dispute	Card disputes
hardcount_creditnote_pending_issue_drafts growing over days	A charge’s result is unclear: clawback drafts
hardcount_audit_write_errors_total rising	Audit log gaps
hardcount_audit_uncovered_mutation_total rising	Audit log gaps
hardcount_billing_failing_subscriptions above zero, or a subscription is not being invoiced	Subscriptions whose billing is failing
A period ended but its invoice is not built yet, or usage is missing from an invoice	Invoice timing and late usage
hardcount_usage_unpriced_quantity_total rising, or a WARN with “matched no price” and “not billed”	Usage with no price
“Usage below zero billed as zero” on a subscription’s timeline, or a WARN with “usage below zero”	Usage below zero
Write I/O high for minutes after a bulk load	Write I/O after a large backfill
p99 jumps, p50 flat, after a quiet spell (RDS)	Commits stall after a quiet spell on RDS
p50 lifts for everyone for seconds on a large table	Autovacuum stalls every commit

Reference

• Health checks, alerts and logs: the health endpoints, which metrics should page someone, and the log lines to search.
• Manual fixes: resolving a dunning run, marking an invoice paid, and a customer with no usable card.
• Escalation: what to do for SEV-1, SEV-2 and SEV-3 incidents.
• Failure-mode entries: infrastructure failures diagnosed and reproduced on real hardware, each with its signature, fix and reproduction. One entry so far (RFM-001).