Pre-1.0 · one maintainer · Stripe and cards only · no SDKs (client libraries) yet · no production users yet. Docs describe v0.9.1 (29 Oct 2026) · What’s new →

HA readiness — SPOF register and the multi-replica plan (2026-07-06)

Record of 2026-07-06, with later amendments. To run two or more copies today, read High availability.

Amendment 2026-10-17. One VM with the Compose stack is also a supported production setup, for a small team that can accept a few minutes of downtime during a restart. Two or more replicas is the high-availability setup. Self-host Hardcount is the current answer.

Amendment 2026-08-30 — the premise below is retired. The supported production posture is now N ≥ 2 hardcount-api replicas behind a load balancer, on managed Postgres with failover + PITR and managed Redis, optionally behind a transaction-mode pooler / RDS Proxy. The trigger that gated the multi-replica build (“a design-partner pilot approaching production cutover”) is treated as fired by decision, not by event; every item in the ordered work-item table below is being re-decided under that posture in the same session. What this does NOT change yet: the code. Until the leader-lease arc (ADR-114, in flight) replaces the session-advisory-lock gate, transaction-mode poolers stay unsupported and the server still refuses to boot behind them — that refusal is the truth of the shipped binary, not of the posture. Single-instance remains a supported deployment size; it is no longer the design assumption.

Design of record for the leader-lease arc: ADR-114 (2026-08-30). Singleton roles (billing, dunning, webhook_outbox, email_outbox, webhook_delivery) become tick-scoped rows in leader_leases, acquired, renewed and released by single self-contained statements on the database clock, with a fencing token every leader-only claim statement proves in its own snapshot. Consequences for this document: hazards 1, 4, 5, 6 and 11 and work items 0, 4, 5, 7 and 8 are re-decided below, in place, each tagged [ADR-114]. The advisory-lock line-cites in this file describe the binary until PR-D of that arc merges; they are history after it.

Single-instance is the supported v1 deployment shape: one hardcount-api process, one Postgres, no load balancer, “~minutes of downtime per deploy/restart is acceptable” (Self-host Hardcount, Operational posture, lines 55-66). Money paths are crash-safe by construction — invoice generation is idempotency-index exactly-once (internal/subscription/postgres.go:1089-1094), dunning transitions are exactly-once CAS (#325), outbound email/webhooks are durable Postgres outboxes with lease claims, and every scheduler sweep re-derives eligibility from durable state — so process death costs latency, not correctness. What single-instance does NOT give you is redundancy, and what N=2 does not yet give you is safety on a handful of named surfaces. This document is the pre-work for the multi-replica build: every single point of failure, every verified hazard of running two API replicas today, and the ordered work list — so that when a design-partner pilot approaches production cutover (the named trigger, docs/self-host.md:67-69), the build is execution, not discovery. Every claim below carries the file:line it was verified against; surfaces that are already safe say so explicitly.

SPOF register

Legend: Lost = irrecoverable without external action. Delayed = self-heals on restart with zero operator work.

Tier: ingest edge

Component	Failure	Blast radius	Lost vs delayed	Operator mitigation today
Single API process — the only front door for usage. All three ingest doors (`POST /v1/usage-events`, `/batch`, `/v1/integrations/litellm/spend`) are synchronous handlers writing straight to Postgres in-request; no Hardcount-side queue/WAL/buffer exists (usage/handler.go:146-167; internal/integrations/litellm/handler.go:104-157; internal/usage/postgres.go:110-117)	Process down = connection refused at the edge; Postgres down with process up = failure per-door (see Downtime = money)	Every usage event fired during the window is refused. Nothing already accepted is ever lost — writes are synchronous, so 2xx means committed (usage/postgres.go:110-117)	Split: delayed for clients that retry non-2xx; permanently lost for clients that don’t — including LiteLLM at its defaults (see Downtime = money). Loss is invisible afterward	Graceful shutdown drains in-flight requests 30s (cmd/hardcount/main.go:294-302) so planned deploys yield clean retriable refusals. Keep deploy windows short, away from tenants’ period close; require ingest clients to retry with idempotency keys

Tier: scheduler / periodic jobs

Component	Failure	Blast radius	Lost vs delayed	Operator mitigation today
Single in-process scheduler goroutine (cmd/hardcount/main.go:237-241), 1h prod tick hardcoded — no config knob; 5m only when `Env=="local"` (main.go:156-159). No tick at boot: first fire is one full interval after start (internal/platform/scheduler/runner.go:55-73)	Process down ⇒ no invoice generation, dunning advances, auto-charge retries, tax/clawback re-drives, trial flips, pause resumes, credit expiry, threshold hard-cap invoices	Outage + up to 1h post-restart silence, then batch-rate drain: one 50-sub batch per livemode per hourly tick, no drain loop in `RunCycle` (internal/billing/engine.go:1697-1740; scheduler.go:327-402 pass `s.batch` everywhere) — backlog recovery latency scales linearly	Delayed only. Every sweep re-derives from durable state (`next_billing_at` in past, `auto_charge_pending`, `next_action_at`); `billSubscription` loops periods per sub so no period is skipped (engine.go:1742-1759)	Supervise with restart=always. For a known backlog: `POST /v1/billing/run` per tenant fully drains that tenant in one call — `RunCycleForTenant` already has the drain-until-empty loop (engine.go:1626-1695); safe to run during a leader tick (ADR-115: every period close proves the period snapshot it read under the subscription row lock, so a drain and the leader’s close can never both bill a period — a subscription the leader is committing at that instant is skipped by the drain’s `SKIP LOCKED` fetch and billed by the leader). [ADR-119, 2026-10-12] “Fully drains” no longer covers a subscription whose billing is failing: the drain skips one that is waiting for its next attempt or is parked. Retry those one at a time with `POST /v1/subscriptions/{id}/retry-billing`. `/health/ready` tracks scheduler liveness (main.go:196-197)

Tier: outbound dispatchers (email outbox, webhook outbox, webhook retry)

Component	Failure	Blast radius	Lost vs delayed	Operator mitigation today
All four background workers run only as goroutines inside `cmd/hardcount` — no separate worker deployable (cmd/hardcount/main.go:238-284)	Process down halts ALL outbound email (invoices, receipts, dunning, password resets, invites — internal/email/dispatcher.go:160-191) and every outbound webhook	Zero outbound delivery tenant-wide for the outage duration; backlog drains on restart at tick rates (email 5 rows/5s; webhook 25 rows/2s — email/dispatcher.go:79-88, webhook/dispatcher.go:53-58)	Delayed only. Queues are durable Postgres tables; rows stay `pending` and are claimed on restart (email/outbox.go:264-268; webhook/outbox.go:179-184; webhook/postgres.go:727-737). One decay case: time-boxed links (1h password-reset tokens) delivered after a >1h outage arrive dead — user re-requests	Process supervision. Monitor PendingCount/FailedCount gauges (email/outbox.go:407-421, 536-550; webhook/outbox.go:326-359). This tier needs zero code for HA — a second replica is already safe (see verified-safe)

Tier: API state (Redis)

Component	Failure	Blast radius	Lost vs delayed	Operator mitigation today
Redis — in production the general and hosted-invoice rate limiters are fail-closed (internal/api/router.go:1087-1089, 1118-1120); nil client or any Redis error ⇒ 429 for every covered request (middleware/ratelimit.go:143-159)	Redis down or `REDIS_URL` unset in prod: operator/dashboard API 429s — login EXCLUDED as of this audit’s PR (/v1/auth previously mounted the fail-closed general limiter, contradicting the stated #21 design; it now has its own never-fail-closed instance); hosted invoice pay pages, payment-update links, cost dashboards 429 (router.go:1216, 1240). Health/metrics exempt (ratelimit.go:83) so an LB thinks replicas are healthy. Still working through the outage: usage ingest (never fail-closed, router.go:1101-1106), inbound Stripe webhooks (no limiter at all, router.go:1204-1207), all background workers	Delayed only — requests fail loudly, clients retry, invoices remain payable; recovery instant when Redis returns. Trap: booting prod without `REDIS_URL` is a mostly-dead API (login + ingest survive) — the misleading “fail open” boot logs were corrected in this audit’s PR	Treat `REDIS_URL` as required in prod; run managed/replicated Redis (tiny — limiters + login counters are its only jobs, verified census: 3 non-test files import Redis); alert on 429 rate and the “failing closed” slog.Error (ratelimit.go:145, 154)

Tier: datastore (Postgres + key material)

Component	Failure	Blast radius	Lost vs delayed	Operator mitigation today
One Postgres holds every piece of state — money tables, outboxes, idempotency keys, sessions, audit log. Boot exits if the DB is unreachable, schema behind, or topology forbidden (cmd/hardcount/main.go:115-152). Hard topology requirement: session-mode PgBouncer or direct connections ONLY — transaction-mode poolers strand the advisory-lock leader gates and are refused at boot by `VerifyAdvisoryLockTopology` (advisory_lock.go:123-178; docs/ops/postgres-requirements.md:87-113)	Postgres down = total outage: every API request fails, `/health/ready` fails, all periodic work stops	Total, for the outage duration	Delayed only inside Hardcount — all pending work is durable rows. Restore-to-earlier-point is the loss case: open-period `usage_events` are Tier-1 (snapshot-only backups lose the unbilled cycle) and already-dispatched outbox rows re-fire as duplicates (docs/ops/backup-considerations.md, with the mark-dispatched UPDATE escape hatch)	Managed Postgres with failover + PITR (the docs’ own recommendation, backup-considerations.md:69-76); quarterly restore drills (prescribed); alert on `/health/ready`
`HARDCOUNT_ENCRYPTION_KEY`	Key lost	Every encrypted column in every backup — PII, webhook secrets, Stripe credentials — reads as garbage	Permanently lost if the key is not backed up independently of the DB (docs/ops/backup-considerations.md)	Store in a KMS/secret manager backed up separately (already documented)

Tier: boot / deploy

Component	Failure	Blast radius	Lost vs delayed	Operator mitigation today
Boot-time migration path (`RUN_MIGRATIONS_ON_BOOT`) — a fleet-wide serialized actor. `applyNoTx` marks `schema_migrations` dirty BEFORE executing (migrate.go:457-459); a mid-migration crash leaves `dirty=true` and `CheckSchemaReady` then refuses boot on EVERY replica (migrate.go:1067-1069). No `migrate force`/`resolve` CLI exists (main.go:57-68 offers status/rollback only; rollback also refuses on dirty, migrate.go:916-918). Losers poll the hybrid lock up to 30min BEFORE the HTTP listener binds (migrate.go:70-73, 236-256 vs main.go:126/288)	Crashed migration = entire API down until a human hand-edits `schema_migrations`; a merely-slow migration stalls the rollout up to 30min per loser attempt (startup probes kill-loop the losers)	Fleet-wide boot refusal; duration = human response time	Delayed only — no data loss; fail-closed is correct here	Run migrations as a dedicated pre-rollout step (`hardcount migrate` in CI / a K8s Job); keep `RUN_MIGRATIONS_ON_BOOT=false` in prod (the default, docs/self-host.md:139); rehearse slow migrations with `cmd/hardcount-migrate-safety`

What breaks at N=2 today

Verified hazards only, ordered by severity. Everything not listed here was checked and is in the verified-safe list below.

Hazards

1. [SHIPPED 2026-07-06 — per-invoice CAS claim, migration 0141] The auto-charge sweep was the one lock-gated money path with no row claim — a Postgres-failover dual-leader window could double-charge one invoice. Fix landed as ClaimAutoCharge (dunning-shaped CAS re-asserting the full eligibility predicate, 5m lease, DB-side now(), updated_at untouched for idempotency-key stability) at the top of the processAutoCharge loop, with release-on-skip for the two provably-pre-Stripe paths. Deliberately NOT claim-in-list: a batch-stamped lease cannot cover a serial 50-invoice loop, and the list methods are shared with dunning enrollment, which a claiming list would starve (adversarial review). Pinned by: real-Postgres claim collision, dual-leader charges-once, predicate-recheck matrix, updated_at-stability, claim-blind lists, release-on-skip tests. Original finding: A failover releases session advisory locks while the old leader’s tick keeps running: the lock pins one conn (advisory_lock.go:44-47, 62-79), tick work runs on other pool conns that reconnect to the new primary, the scheduler never rechecks lock liveness mid-tick (scheduler.go:271-282), and Release() swallows errors (advisory_lock.go:90). During the overlap, every duplicated step collapses via SKIP LOCKED / CAS / idempotency-index — except the auto-charge sweep: ListAutoChargePending is a plain SELECT with no claim (invoice/postgres.go:1813-1827), and its only dedupe is the Stripe idempotency key derived from inv.UpdatedAt (payment/stripe.go:451-484). The divergent-key path: leader A’s charge outcome persists as failed/unknown via UpdatePayment (bumps updated_at, status stays finalized, invoice/postgres.go:703-709); leader B’s refresh (engine.go:5339-5344) then mints a NEW key — chargeInvoice checks only Status and amount_due, never payment_status (stripe.go:409-426). Double charge fires when A’s outcome was UNKNOWN but the PI actually succeeded. Rare (failover coinciding with a tick), customer-visible, refundable, never silent (both PIs emit webhooks). Fix: give the sweep the dunning-shaped CAS claim (a bare FOR UPDATE SKIP LOCKED is insufficient — the list tx ends before the charge loop). [ADR-114] The mechanism behind the original finding — a session lock pinned to one connection while the tick ran on others — no longer exists; leadership is a row, and a Postgres failover leaves exactly one token valid. The per-invoice CAS claim stays as the money guarantee.

2. Rolling-deploy binary skew permanently DLQs money emails of a new email_type. [SHIPPED 2026-08-30 — unknown email_type is ErrUnknownEmailType, retryable and attempt-capped; a newer replica delivers it, a type no binary knows still DLQs at the cap.] email_outbox.email_type deliberately has no CHECK constraint (outbox_sender.go:40-43, exercised by ADR-082’s TypeCreditNote), the old binary’s dispatcher returns ErrPayloadDecode for unknown types (dispatcher.go:189-190), which is classified permanent (outbox.go:81-87) and DLQ’d with zero retries (outbox.go:338-343). The per-tick dispatcher lock is won arbitrarily by either binary (dispatcher.go:118-127), so new-type rows claimed by the OLD binary in the overlap window are dead — not delayed. A per-row slog.Error fires (outbox.go:344-351) but FailedCount has zero callers and no redrive tooling exists (verified by grep); recovery is hand-written SQL. Inert at N=1. Fix (shipped): unknown type ⇒ retryable backoff.

3. [AMENDED 2026-08-30 — the migrate.go comment clause below was fixed by #395 on 2026-07-06 (the code now WARNs and continues and says so honestly); the expand/contract discipline, the DDL lint and ADR-116/117 remain open as program Phase 3] Old-code-on-new-schema windows are governed by nothing. When dbVer > embedded, CheckSchemaReady WARNs and returns nil while its own inline comment says it should refuse (migrate.go:1075-1090; the exported godoc at 1045-1055 promises nothing either way). Leader election is version-blind — fixed lock constants, no version affinity (main.go:193) — so during a rolling deploy the OLD replica’s scheduler/dispatchers can keep winning the locks and run old billing/dunning code against the new schema for the whole window (unbounded if the rollout wedges at 1-old/1-new). No expand/contract migration discipline exists; cmd/hardcount-migrate-safety measures lock holds and duration, not old-binary compatibility (its main.go:1-37). Worst case is silent data mis-writes, not downtime. Fix: decide + document the skew contract, fix the comment, optionally lint destructive DDL.

4. [AMENDED 2026-08-30 — ADR-114: the lock below is now a lease the wedged tick keeps renewing (its heartbeat goroutine is alive even when the work is stuck), so the effect is unchanged: the role stays held; hardcount_leader_last_tick_age_seconds growing while hardcount_leader_held stays 1 is the cluster-wide signal this hazard lacked] Health model masks a wedged scheduler leader at N=2. Ticks have no per-tick timeout (runner.go:55-90) and each advisory lock lives on a dedicated conn released only by Release() or session drop (advisory_lock.go:62-92) — a tick wedged mid-half holds that half’s lock indefinitely. The wedged pod’s /health/ready goes 503 (LB pulls it) but K8s liveness /health is an unconditional 200 (router.go:1172-1173, 1447-1449), so nothing restarts it; followers report ready because onRun fires on lock-skipped ticks (scheduler.go:249-254). Billing and dunning are separate lock keys (scheduler.go:271-282 vs 485-496), so exactly ONE half stops cluster-wide; the per-pod staleness metric still fires for the pulled pod (runbook.md:32) — alertable, not silent, but only if metrics alerting is deployed. Related N=1 hazard: readiness couples traffic to scheduler pace — a steady-state tick gap >2× interval 503s the sole replica’s readiness and self-inflicts a full API outage (router.go:1474-1481; runbook.md:11-12 wires it as the LB check). A restart’s long first tick does NOT trigger it (zero lastRun ⇒ “ok: awaiting first run”, router.go:1482-1483). Fix: per-half tick timeout; decouple readiness from scheduler staleness; make scheduler liveness a cluster fact persisted in the DB. [ADR-114] Cluster-visible half shipped by the arc: leader_status + hardcount_leader_last_tick_age_seconds{role}. The per-role tick deadline and the readiness swap remain open (work item 5). [2026-10-13] The readiness swap shipped: a stale tick is reported in the /health/ready body and no longer changes the status code, so the wedged pod is not pulled and the N=1 hazard above is closed. The per-role tick deadline is still open.

5. [CLOSED 2026-08-30 — ADR-114 fence (cutover PR-D) + the two CAS fixes shipped in PR-B (ClearPauseCollection CAS, watermark CAS); the effect-firer audit is recorded as ADR-114’s first amendment: every effect reachable from a leader tick sits behind a conditional write] Advisory-lock leadership has no fencing: dual-leader overlap duplicates non-CAS side effects (outbound webhook events). Same failover mechanism as hazard 1; money converges everywhere, but effect-firing tied to idempotent-but-not-CAS writes duplicates — e.g. pause auto-resume: the SKIP LOCKED list tx rolls back before the clear (subscription/postgres.go:1211-1252), ClearPauseCollection is unconditionally idempotent with no changed-rows signal (postgres.go:643-679), and each overlapping tick fires subscription.collection_resumed via a fresh outbox enqueue (service.go:2337-2358; outbox.go:97-117). The duplicates carry DIFFERENT envelope ids (separate Enqueue calls, distinct timestamps), so the documented dedupe-on-event-id guidance (docs/webhooks.md:126) does NOT collapse them — consumers see true semantic duplicates. Rare window; fix is a cheap lock-liveness probe (SELECT 1 on the lock conn between phases, abort tick on error) rather than full fencing tokens. [ADR-114] A leader that has been superseded, expired or paused admits zero rows from the first claim statement that starts after the change committed (leader_fence() in the five claim funnels), and its work ctx is cancelled within one heartbeat. The one named non-CAS effect-firer (ClearPauseCollection → subscription.collection_resumed) becomes a CAS. Residual: outbound events remain at-least-once across a takeover — a frozen process that resumes may complete one already-started statement.

6. [MOOT 2026-08-30 — ADR-114 deletes VerifyAdvisoryLockTopology in PR-D] Topology self-check has a contention false-pass under restart loops. VerifyAdvisoryLockTopology returns nil when its probe lock is already held, assuming a concurrent booter (“its passing verdict covers this topology too”, advisory_lock.go:139-143). Behind a transaction-mode pooler, a first boot that acquires the probe lock and then fails a later probe strands it (the PID-flip path at :145-150 errors without unlocking); every subsequent boot inside the strand window (systemd restart loop, second replica) sees got=false ⇒ nil ⇒ starts behind the forbidden pooler, where all five worker locks then strand silently — the exact “week of missing invoices with zero errors” the check exists to prevent (main.go:144-148). Window is minutes-to-an-hour per strand on stock PgBouncer timeouts, but restart loops retry in seconds and the effect persists as long as the topology does. Fix: retry acquisition 3×200ms and fail boot if still contended (~10 lines).

7. [AMENDED 2026-08-30 — shape (b) has a DESIGNED trigger since #864: a surge-first rolling deploy abandons the last old replica’s in-flight advance after 30 s and no successor boots afterwards, so the clock stays advancing until ANY replica restarts; runbook §6, deploy/README and the shutdown log line used to promise “resumes at the next boot” — corrected same day. Real fix unchanged: program ha-9 / ADR-118] Test-clock catchup is the one scheduler-adjacent worker with NO leader gate. The queue is an in-process channel, self-declared single-process (catchup.go:63-67: “swap for a Redis or Postgres LISTEN/NOTIFY queue”), and RecoverInFlight runs ONCE at boot, re-enqueuing every status='advancing' clock with no lease/ownership check (main.go:254-261; testclock/service.go:760-778). Two N=2 shapes: (a) rolling deploy — replica B boots mid-catchup and both run RunCatchup concurrently for the same clock (recovery bypasses the MarkAdvancing CAS; no lock anywhere in the path). Duplicate invoices CANNOT result (idempotency index; testclock transitions are CAS, postgres.go:224-269) — damage is duplicate test-mode webhook/timeline noise and a possible spurious internal_failure, recoverable via Retry. (b) Inverted liveness — the owning replica crashes, the survivor never rescans, and the clock wedges at ‘advancing’, which has no NON-DESTRUCTIVE exit (Advance needs ‘ready’, Retry needs ‘internal_failure’; Delete works but cascade-cancels pinned subs, postgres.go:129-162). At N=1 restart heals this; at N=2 it’s worse. Test-mode only — clock-pinned subs are excluded from all wall-clock paths (subscription/postgres.go:1048-1056). Fix: DB-backed claim (CAS lease or per-clock advisory lock) + periodic rescan on a leader-gated tick; the CatchupQueue interface was built for the swap (catchup.go:30-43).

8. Webhook SSE live tail is a per-process EventBus — at N=2 the dashboard sees a partial, misleadingly-alive tail. In-process pub/sub, no cross-replica fan-out (eventbus.go:86-160). Frames originate on BOTH replicas: HTTP-path dispatches on whichever replica served the request (payment/settlement.go:436; subscription/handler.go:394; webhook/service.go:704) and dispatcher/retry frames on whichever won that tick’s try-lock — not a stable leader — so session affinity does not fully mitigate. The connect-time 50-event snapshot is DB-backed (sse.go:84-92); refresh shows truth. UI-only, no delivery impact. The code names the fix itself: Postgres LISTEN/NOTIFY (sse.go:33-41). Safely deferrable past first cutover.

9. [SHIPPED 2026-08-31 — the cap is derived from password_reset_tokens (count per account per hour, decided inside the insert’s transaction under a user-row lock): one budget fleet-wide with NO external dependency. It went via a Redis bucket (ha-14 PR-B) for a few hours; a walk showed that cap vanished wherever Redis was unreachable, so the durable rows became the budget. Keyed on the account, so the capped reply is byte-identical to the unknown-address reply] Password-reset per-address throttle is per-process: cap degrades 3 → 3N/hour. In-process map, comment explicitly scoped to single-process deploys (user/handler.go:46-48, 51-81, enforced at :353). No enumeration or auth impact; the Redis-backed per-IP limiter still bounds a single attacker fleet-wide. Fix: reuse the Redis GCRA limiter shape (internal/api/middleware/ratelimit.go) — the previously named FallbackFailureCounter was deleted by #497 [done 2026-08-31], ~half day, deferrable.

10. Threshold “crossed-but-deferred” notes dedup is a per-process sync.Map: exactly one duplicate audit/timeline row per (sub, period) at N=2. Per-tick lock release means replicas alternate ticks in normal operation; each replica’s map emits the note once, capping duplicates at N−1 per (sub, period) plus one per restart (engine.go:79-81; threshold_scan.go:363-371; scheduler.go:271-282). Operator-visibility artifacts only — the billing decision is invoice-line-watermark-based (threshold_scan.go:316-358). Accept, or dedup against the audit table (~2h).

11. [SHIPPED 2026-07-06 — ClaimChargeForDunningRetry, same lease column] Sweep vs dunning retry could double-charge via purpose-suffixed idempotency keys. Fix: the dunning retry path takes the shared per-invoice lease before charging (predicate payment_status IN ('pending','failed') — ‘unknown’ deliberately excluded so ambiguous outcomes wait for the reconciler, which also closes the same-tick N=1 window: the billing half’s unknown outcome followed seconds later by the dunning half’s due retry); claim-lose maps to ErrTransientSkip (no attempt burn). Pinned by cross-path mutual-exclusion + status-matrix tests. Original finding: The billing and dunning scheduler halves run under SEPARATE advisory locks, so two replicas run them concurrently by design (scheduler.go:218-227). Reachable overlap: a card-less pending invoice enrolled by EnrollStalledForDunning keeps auto_charge_pending=TRUE + payment_status=‘pending’ + an active run; the customer attaches a card; next cycle the auto-charge sweep (purpose “”) and a due dunning retry (purpose “dunning_retry”) can charge concurrently — the purpose suffix makes their Stripe idempotency keys differ BY CONSTRUCTION (stripe.go:474-477), so Stripe cannot dedupe. Sequential orderings are safe (inline SettleSucceeded → paid → the other path’s Finalized guard refuses). Fix shape: dunning’s RetryPayment takes the same per-invoice claim CAS (predicate widened to payment_status IN (‘pending’,‘failed’)) and maps claim-lose to a transient skip (no attempt burn). Needs its own dunning-side tests. [ADR-114] Unchanged: the shared lease column is the guarantee and is deliberately NOT fenced (ADR-114 §Alternatives).

Verified safe at N=2 — do not re-flag

Ingest dedup: fully DB-level — ON CONFLICT (tenant_id, livemode, idempotency_key) DO NOTHING; both doors translate the loser to success (usage/postgres.go:113-115, 169, 182-186; usage/handler.go:171-187; litellm/handler.go:183-197). Batch is one all-or-nothing tx (usage/postgres.go:44-78).
Rate limiting with healthy Redis: shared GCRA buckets, per-limiter namespaces — global limits, no per-replica doubling (ratelimit.go:19-37, 171-208; router.go:1077-1128).
API Idempotency-Key middleware: Postgres-backed reserve-before-act; racing replicas resolve one winner, the loser polls the DB for the winner’s response; DB-error fail direction is 409-retry, never re-execute (middleware/idempotency.go:102-131, 203-234, 298-323).
Dashboard sessions: fully DB-backed incl. cross-replica revocation; nothing session-shaped in process memory (session/store.go:42-136).
Scheduler lock coverage is COMPLETE: every scheduler job — 7 reconcilers, auto-charge retry, dunning enrollment, threshold scan, trial expiry, pause resume, RunCycle, credit expiry, cross-mode cleanup — runs inside exactly one of the two advisory-lock scopes (scheduler.go:268-299, 480-508; main.go:193). Followers stay health-green (onRun fires on skipped ticks, scheduler.go:249-254).
Invoice generation is exactly-once independent of the leader lock: idx_invoices_billing_idempotency, loser gets 23505 and skips gracefully (subscription/postgres.go:1089-1094).
Trial expiry / pause resume double-processing: SKIP LOCKED candidate fetch with explicit multi-replica rationale + CAS status flips (subscription/postgres.go:2004-2008, 2067-2090, 735/832/886). Credit expiry converges (ExpireGrantAtomic, credit/service.go:469-500).
All three drainers (email outbox, webhook outbox, webhook retry) (amended 2026-08-30: the two tests named below cover the email outbox and the webhook RETRY claim; the webhook-OUTBOX drainer’s own claim was held by convention until TestOutbox_ProcessBatch_SkipsRowsLockedBySibling (pins FOR UPDATE SKIP LOCKED) and TestOutbox_ProcessBatch_ConcurrentClaimersDisjoint (pins the claim lease) — each mutation-verified to go red for exactly its own mechanism.): two-layer design — per-tick advisory-lock gating for efficiency, claim-lease + FOR UPDATE SKIP LOCKED + CAS status marks for correctness, disjointness pinned by tests (TestP5_ConcurrentClaimersDisjoint, TestListPendingDeliveries_ConcurrentClaimersDisjoint). Lease claim atomically bumps attempts, dead claimers’ rows re-claimable after lease expiry, marks are CAS’d so a zombie replica can never cause a third send. Webhook event + deliveries + outbox-mark commit in ONE tx (handler-owns-mark), so no crash can mint a duplicate event id (webhook/postgres.go:375-452; email/outbox.go:259-272, 363-391).
Livemode/RLS propagation on ctx-detached workers: both dispatchers pin the row’s livemode per-row; scheduler halves fan out per livemode with asserts (email/dispatcher.go:153-158; webhook/dispatcher.go:126-127; scheduler.go:287-310) — pinned by dispatcher_livemode_test.go (d526e94).
Concurrent-replica migrations: the hybrid loop serializes on LockKeyMigrateHybrid on a dedicated 1-conn pool; losers poll, re-read schema_migrations under the lock, skip applied work, and refuse to step past a dirty peer (migrate.go:125-150, 202-338).
Boot gauntlet before listener bind: migrations → CheckSchemaReady → topology check → ADR-073 RLS posture checks; a misconfigured replica cannot join the fleet (main.go:122-152 vs :288).
Leader crash mid-tick: session advisory locks auto-release on TCP session drop — no zombie-lock recovery needed (advisory_lock.go:58-61); panicking ticks are recover()’d and cannot kill any worker goroutine (runner.go leadOneTick; since 2026-10-08 a panicked tick is also released as not completed, ADR-114 amendment).
Login brute-force protection: no per-account login state exists anywhere (ADR-094 v1, #497/#498 removed lockout.go and the interim Postgres throttle); the per-IP Redis limiter on /v1/auth (100/min) is the only bound, shared by all replicas — and it is only per-IP when TRUST_PROXY covers the ingress (sweep 2026-08-30 S3). (Rewritten 2026-08-30: the previous text cited a FallbackFailureCounter deleted on 2026-07-17.)
Stripe circuit breaker: per-process but fails safe — breaker-rejected calls are transient/retryable, no invoice failed, no dunning tick; worst case N× half-open probes (payment/breaker/breaker.go:48-56, 89-93).
Webhook outbox payload binary-skew: the dispatcher never interprets the payload — opaque pass-through, old binary delivers new payloads unchanged (webhook/dispatcher.go:118-133). Email cc FIELD skew-safe by design (outbox_sender.go:47-56) — the type TAG was hazard 2, closed 2026-08-30.
Remaining in-process state is benign: log dedupers, Prometheus gauge-registration guards, test-only mutexes (postgres.go:111; router.go:61). Per-replica metrics counters are standard scrape-and-aggregate.
One comment-only drift for the HA PR: cmd/hardcount/main.go:266 says “LockKeyWebhookDispatcher”; the constant is LockKeyOutboxDispatcher (advisory_lock.go:17). Cosmetic.

Downtime = money?

The honest answer: inside Hardcount, nothing is ever lost — the entire loss surface is the ingest edge, and it is decided by client retry behavior. Writes are synchronous, so 2xx means committed; everything pending (billing cycles, dunning steps, auto-charges, outbox rows) is durable and self-heals on restart (docs/ops/backup-considerations.md documents the full resume ledger with validation queries). Inbound Stripe webhooks 5xx during an outage and Stripe retries for ~days, with the payment_unknown reconciler as backstop. What does NOT self-heal:

1. LiteLLM usage fired while Hardcount is down is permanently lost at LiteLLM’s defaults — this was an open question and is now source-verified (BerriAI/litellm main, read 2026-07-06). LiteLLM’s generic_api logger defaults max_retries=0 and its async_send_batch ends with finally: self.log_queue.clear() — on ANY send exception the batch is dropped, no re-queue, no durable buffer. Consequences: (a) the repo’s three claims that “LiteLLM retries (only) on 5xx” (docs/adr/033-litellm-spend-adapter.md; router.go:1093-1096; litellm/handler.go:104-107) are false at current LiteLLM defaults — 5xx raises via raise_for_status() into the same drop path, so Postgres-down loses events exactly like process-down; (b) retries are operator-configurable (max_retries/retry_delay), which shrinks but never closes the window; (c) a LiteLLM-proxy restart loses its in-memory log_queue even while Hardcount is healthy. Two Hardcount-side aggravators verified in-repo: during a full Postgres outage the request dies at API-key auth, which masks DB failure as 401 (auth/middleware.go:37-47, ADR-026 anti-enumeration) — not a retryable status under any config; and on partial DB failure the spend handler appends persist errors to the envelope and returns 200 OK unconditionally (litellm/handler.go:138-143, 156) — the comments claiming storage errors propagate as 5xx describe code that does not exist. At the rate-limit sizing floor (>1.7 calls/s, router.go:1094), a 2-minute deploy is ~200+ silently unbilled calls.

2. The recovery path exists and is clean — but has a hard deadline. Deterministic per-call idempotency keys (<litellm_id>:input/:cache_read/:output, mapper.go:220-243), DB-level ON CONFLICT DO NOTHING, and silent-success replay make operator re-POST of LiteLLM’s own spend logs a pure gap-filling replay with preserved original timestamps (mapper.go:139-148). The deadline: by documented design (usage/service.go:214-226, “KNOWN BEHAVIOR”), an event whose timestamp falls in an already-finalized period is stored but effectively unbilled — aggregation is strictly timestamp-windowed (usage/postgres.go:344-393) and nothing true-ups closed periods. Replay after finalize preserves the audit ledger but not the revenue; recovery then means a manual invoice adjustment. A downtime window straddling period close, discovered late, is the permanent-loss case. The late-event counter named in that comment is the smallest visibility fix.

3. Direct API clients: delayed if they retry non-2xx, lost if they don’t. The retry contract (“non-2xx means not recorded — retry; replays are dedup-safe”) is not yet stated in the ingest docs. One narrow pre-existing N=1 hazard, N=2-neutral: a keyless single-event POST whose response died to a hard crash (kill -9 between commit and response; graceful deploys drain 30s) double-bills on retry — keyless events have “no dedup line of defense” (usage/postgres.go:48), NULLs never conflict. Batch is immune even keyless (one tx); LiteLLM is immune (always keyed). Docs-first fix; do not reject keyless (Stripe meter events accept keyless too).

4. Shutdown mechanics (SHIPPED 2026-08-30 — three-stage 90s budget: HTTP 30s → catch-up 30s → workers 30s; readiness-first drain with SHUTDOWN_DRAIN_DELAY; SSE subscribers closed on shutdown; compose grace 120s. The paragraph below is the pre-fix analysis.): worst-case drain is ~11min (HTTP 30s → test-clock catchup up to 10min on a context.Background() ctx that shutdown cannot cut short, catchup.go:57/147, main.go:294-328 — the comment at main.go:308-309 claiming otherwise is wrong → workers 30s), exceeding K8s’ default 30s grace; SIGKILL mid-anything is designed-for (outbox rows pending, catchup resumes at boot). Set grace ≥ ~75s and cap the shutdown catchup wait. Unclean kills also cost bounded at-least-once duplicates: ≤5 emails + ≤10 webhook POSTs re-sent once each, webhooks with the SAME event id (receivers dedupe; SMTP can’t), and one of 15 attempts burned per claimed in-flight row — deliberate, so a crash-looping row still converges to the DLQ (email/outbox.go:189-196; webhook/postgres.go:436-446).

5. Restore-to-earlier-point and key loss (see SPOF register): duplicate outbox re-fires + lost open-period usage unless PITR; a DB backup without HARDCOUNT_ENCRYPTION_KEY is unrecoverable.

The scoped multi-replica build

Named trigger: a design-partner pilot approaching production cutover (docs/self-host.md:67-69). Until then, single-instance + supervision + PITR is the supported posture. (Amended 2026-08-30: trigger treated as fired — see the header. The table below is re-decided under the N ≥ 2 posture; item 0’s “keep session-mode pooling only” is superseded by the leader-lease arc, which makes the app pool pooler-safe by construction.)

Already exists (build on, don’t rediscover): leader-lease gating — per-tick lease row + fence token re-checked in every claim funnel (ADR-114, internal/platform/leader; until 2026-08-30 this was session advisory-lock gating) — wired for scheduler + all three drainers; SKIP LOCKED + lease + CAS claims verified disjoint by tests; Postgres-backed sessions and idempotency; migration serialization on LockKeyMigrateHybrid; (VerifyAdvisoryLockTopology retired with ADR-114); dunning exactly-once CAS as the claim template; RunCycleForClock/RunCycleForTenant drain loops as the backlog template (engine.go:1506-1598, 1626-1695); FallbackFailureCounter as the Redis-counter template (lockout.go:165-200); the SSE LISTEN/NOTIFY design note (sse.go:33-41); the swappable CatchupQueue interface (catchup.go:30-43).

Ordered work items:

#	Item	Size
0	Ops prerequisites: managed Postgres HA + PITR; managed Redis; LB + health-check wiring incl. `TRUST_PROXY` = the ingress CIDR and `SHUTDOWN_DRAIN_DELAY` ≥ the balancer’s health-check window (sweep 2026-08-30 S3/S17). Keep session-mode pooling only	ops, days
1	Redis truth + boot guard: fix the lying “fail open” messages (config.go:138-139, router.go:1053/1063); refuse prod boot when `rdb==nil`; fix the router.go:1084-1086 auth-limiter comment — SHIPPED 2026-08-30 (production refuses to boot on unset OR unreachable Redis; messages corrected; ADR-073 amended)	done
2	Auto-charge sweep claim (hazard 1) — SHIPPED 2026-07-06 (per-invoice CAS in processAutoCharge, NOT on the list — see hazard 1)	done
3	Email dispatcher skew (hazard 2): unknown `email_type` ⇒ retryable, attempt-capped — SHIPPED 2026-08-30	done
4	Scheduler: immediate first tick at boot (runner.go, ~5 lines); port the drain-until-empty loop into `RunCycle` + batch-capped sweeps	~half day
5	Health/leadership (hazard 4): per-half tick timeout (< interval); readiness = DB ping only; scheduler liveness as a cluster fact (last-successful-leader-run in DB, alertable from any replica)	1-2 days
6	Deploy discipline (hazards 3, migration SPOF): migrations as pre-rollout Job, `RUN_MIGRATIONS_ON_BOOT=false`; add `hardcount migrate force/resolve` (~50 LOC + runbook); decide + document the expand/contract skew contract, fix the migrate.go:1076-1079 comment; optional destructive-DDL lint	small + medium
7	Topology check contention retry (hazard 6): 3×200ms then fail boot	~10 lines + test
8	Lock-liveness probe (hazard 5): `SELECT 1` on the lock conn between tick phases, abort on error; document outbound events as at-least-once	~half day
9	Test-clock catchup (hazard 7): DB-backed CAS lease claim + periodic leader-gated rescan of ‘advancing’	1-2 days; test-mode only — interim: “restart any replica to recover a wedged clock”
10	LiteLLM loss-window shrink: spend handler returns 5xx on real storage errors (make handler.go:107/163-165 comments true); correct ADR-033’s false retry claim; recommend `max_retries>0` in callback config; outage-replay runbook in docs/integrations/litellm.md; decide 503-vs-401 for DB-down at the ingest auth edge — SHIPPED 2026-08-30 (persist edge returns 503 `ingest_unavailable`; ADR-033 amended; retry knobs documented)	done
11	[SHIPPED 2026-08-31 — contract in docs/api-surface.md; ingest 5xx releases the Idempotency-Key (isIngestPath) instead of pinning it; the late-event counter already existed] Ingest docs: retry contract + “idempotency_key strongly recommended for at-least-once pipelines”; late-event counter for finalized-period landings	small
12	Shutdown: cap catchup wait at ~30s (1 line, main.go:310); document grace-period numbers — SHIPPED 2026-08-30 (catch-up wait 30s; 90s budget published; readiness-first drain + SSE close-on-shutdown from the hazards sweep, finding A)	done
13	Tax-reversal promote-on-detect + pending-reversals gauge (ADR-062 amendment 2026-07-06): the CN/invoice tax-reversal sweeps’ structural branch detects marker-less orphans but does not promote them, so the 24h window bounds Stripe recovery, not detection — a >24h tax-API outage coinciding with the compound failure silently over-remits. One local write + gauge + runbook row	~1-2h
14	Deferrable past cutover: SSE LISTEN/NOTIFY fan-out (~1-2 days); reset throttle → Redis (~half day); threshold-note DB dedup (~2h)	as noted

Items 0-2 unlock N=2 for the ingest edge — the tier that most rewards redundancy and needs no leader election (verified stateless). Items 3-9 make rolling deploys and failover honest. Total: roughly two engineer-weeks plus ops.

Refuted-hazard appendix

One line each, so future audits don’t re-flag. (No whole finding was refuted this audit; these are refuted candidate-hazards and refuted sub-claims.)

Test-clock double-run duplicates invoices — refuted: idx_invoices_billing_idempotency (subscription/postgres.go:1089-1095) + CAS clock transitions (testclock/postgres.go:224-269); loser skips gracefully.
Concurrent credit-applies diverge the auto-charge idempotency key — refuted: ApplyToInvoiceAtomic serializes under FOR UPDATE, at most one racing apply bumps updated_at (credit/postgres.go:609-792); the real divergence path is UpdatePayment after a failed/unknown outcome (hazard 1).
“Postgres-down is safer than process-down for LiteLLM” — refuted: _should_retry_exception treats TransportError and 5xx identically; both hit the same drop path.
Fast ticks overlap at N=1 — refuted: ticks are strictly sequential per replica (runner.go:64-73); dual-leader overlap requires N≥2.
Envelope-id dedup absorbs dual-leader duplicate events — refuted: separate Enqueue calls mint distinct ids (webhook/outbox.go:109), so docs/webhooks.md:126 guidance does not collapse them.
A wedged ‘advancing’ clock has no operator exit — overstated: Delete is not status-gated (testclock/postgres.go:129-145) — destructive (cascade-cancels pinned subs) but available.
Restart-with-backlog first tick trips readiness — refuted: zero lastRun reports “ok: awaiting first run” (router.go:1482-1483).
Scheduler leadership “flaps” and re-emits threshold notes per flap — refuted: no sticky leadership exists (per-tick lock); each replica’s sync.Map bounds emission to once per process per (sub, period).
Dashboard sessions / idempotency cache split-brain at N=2 — refuted: both Postgres-backed (session/store.go:42-136; middleware/idempotency.go:203-234).
Rate limits double at N=2 — refuted for all Redis-backed limiters (shared GCRA); true only for the in-process reset throttle (hazard 9).
Outbox dual-drain double-sends — refuted: SKIP LOCKED claims + lease + CAS marks, pinned by concurrent-claimer tests.
Racing boot migrations at N=2 corrupt schema state — refuted: serialized on LockKeyMigrateHybrid with under-lock re-read and dirty refusal (migrate.go:202-338).
Redis is a boot SPOF — refuted: boot warns and continues without Redis (router.go:1049-1059); it is a prod runtime availability SPOF instead (fail-closed limiters).
Batch ingest can commit a prefix and double-bill on retry — refuted: single all-or-nothing tx (usage/postgres.go:44-78).
Follower replicas report degraded scheduler health at N=2 — refuted: onRun fires on lock-skipped ticks (scheduler.go:249-254).
“LiteLLM retries on 5xx” (as the repo claims in three places) — refuted at LiteLLM defaults (max_retries=0, batch dropped on any exception); the repo comments need correcting, not the finding.
Drainer downtime loses queued messages — refuted: durable rows, claim-at-lease, at-least-once with bounded duplicates; only decay case is time-boxed links (>1h outage).

Delta sweep 2026-08-30 (after ADR-114)

Six-lens fan-out over origin/main 940d6789 (per-process state, ungated background work, HTTP layer under a load balancer, database-session assumptions, observability/ops docs, dashboard contract), each finding refuted by two independent agents (code trace; register + earns-its-place) and re-read by the maintainer. 12 confirmed by both refuters, 1 contested, 6 refuted; 8 further findings hand-verified after the refuter pass was interrupted. Everything not listed here was traced and found safe on current main — see “verified again” below.

#	Finding (file:line)	Severity	Class	Disposition
S1	Threshold-reset fire vs cycle close via `POST /v1/billing/run` double-bill usage. The leader’s threshold fire (`threshold_scan.go:794-805`) and an operator-triggered close (`engine.go:3240` invoice, `:3420` advance in a separate tx) are two period writers with disjoint idempotency seams (partial threshold index vs 4-tuple cycle index) and no shared row lock; the reset arm’s `UpdateBillingCycleTx` is `WHERE id` only. Usage in `[P0,t1)` lands on both invoices; in one order the watermark also moves backwards. Reachable at N=1 (HTTP goroutine vs scheduler); silent at N≥2.	money	two unserialized period writers	FIXED 2026-08-30 — ADR-115: every period writer (cycle close, threshold fire on both reset arms, plan swap, scheduled cancel) is `ClosePeriodTx`, one UPDATE that row-locks the subscription and proves the (status, period start, watermark) snapshot it read, as the FIRST statement of the transaction that also inserts the invoice; the close re-reads the threshold watermark in-tx; the swap refuses a period a fire already billed past (409 `subscription_period_moved`). Real-Postgres collision suite incl. a 20-iteration barrier race asserting both lock orders; doctor alarm `usage_billed_once_per_subscription_meter_window`. Ops prerequisite: `idle_in_transaction_session_timeout` on `hardcount_app` (postgres-requirements)
S2	An `Idempotency-Key` reservation orphaned by a hard kill or a handler panic stays `pending` 24 h; every retry — on any replica — polls 5 s and gets 409 “please retry” (`idempotency.go:383`; `Recoverer` is mounted outside the middleware, so a panic orphans the row too).	correctness	reservation without lease (liveness sink, playbook class G)	FIX with program ha-11: stamp `reserved_at`; past the route’s timeout budget answer a distinct 409 “outcome unknown — GET the resource” (releasing for re-execution only after auditing every wrapped handler is transactional)
S3	`TRUST_PROXY` unset behind an LB collapses every per-IP bucket into one (`router.go:1332`): hosted pay pages 60/min for the whole install (~6-10 concurrent payers), logins 100/min shared and lockable by one client; production boots with an Info log.	ux / security	deploy prerequisite without boot guard (ha-1’s class)	FIXED 2026-08-30 — production boot guard (`TRUST_PROXY=none` opt-out); deploy/README wiring section
S4	`hardcount_parked_invoices` is a per-replica `.Set()` gauge stamped only by the replica that led the billing tick (`metrics.go:287`, `reconciler.go:180`); followers export a stale copy of a cluster fact, so the runbook’s “needs a human” alert fires from a replica whose number was resolved hours ago.	ops	cluster fact as per-process gauge	FIXED 2026-08-30 — `GaugeFunc` from the DB per scrape
S5	A self-abandoned tick stamps `last_tick_holder`/`last_tick_ended_at` (`manager.go:87`, release outcome 2) — cadence and observability share one column, so `leader_status`/`hardcount_leader_last_tick_age_seconds` report a completion that never happened.	ops	observability lie	FIX — separate cooldown column (migration 0175; ADR-118’s test-clock role takes 0176)
S6	Boot-time ADR-101 interval backfill has no cross-replica serialization; two replicas booting together race the same INSERTs and the loser `os.Exit(1)`s (`main.go:171`). One extra restart, no partial data (per-partition tx, EXCLUDE constraints).	ops	unserialized boot one-shot	FIXED 2026-08-30 — SQLSTATE 23P01 = “a sibling reconstructed this partition first”; deterministic two-replica test
S2	An `Idempotency-Key` reservation orphaned by a hard kill or a handler panic stays `pending` 24 h; every retry — on any replica — polls 5 s and gets 409 “please retry” (`idempotency.go:383`; `Recoverer` is mounted outside the middleware, so a panic orphans the row too).	correctness	reservation without lease (liveness sink, playbook class G)	FIXED 2026-08-30 — a reservation pending > 5 min answers 409 `conflict_idempotency_unresolved` at once (outcome unknown — GET the resource); never re-executed
S3	`TRUST_PROXY` unset behind an LB collapses every per-IP bucket into one (`router.go:1332`): hosted pay pages 60/min for the whole install (~6-10 concurrent payers), logins 100/min shared and lockable by one client; production boots with an Info log.	ux / security	deploy prerequisite without boot guard (ha-1’s class)	FIX — production refuses to boot unless `TRUST_PROXY` is set or explicitly `none`; deploy/README wiring section (docs done this PR)
S4	`hardcount_parked_invoices` is a per-replica `.Set()` gauge stamped only by the replica that led the billing tick (`metrics.go:287`, `reconciler.go:180`); followers export a stale copy of a cluster fact, so the runbook’s “needs a human” alert fires from a replica whose number was resolved hours ago.	ops	cluster fact as per-process gauge	FIX — `GaugeFunc` from the DB per scrape (the queue-depth pattern)
S5	A self-abandoned tick stamps `last_tick_holder`/`last_tick_ended_at` (`manager.go:87`, release outcome 2) — cadence and observability share one column, so `leader_status`/`hardcount_leader_last_tick_age_seconds` report a completion that never happened.	ops	observability lie	FIXED 2026-08-30 — migration 0175 `not_before`; lost ticks never touch `last_tick_*`
S6	Boot-time ADR-101 interval backfill has no cross-replica serialization; two replicas booting together race the same INSERTs and the loser `os.Exit(1)`s (`main.go:171`). One extra restart, no partial data (per-partition tx, EXCLUDE constraints).	ops	unserialized boot one-shot	FIX — treat SQLSTATE 23P01 as “a sibling reconstructed this partition first”
S7	Rolling deploy strands a test-clock advance (hazard 7 shape (b), now a designed trigger via #864); runbook §6, deploy/README and the shutdown log promised “resumes at the next boot”.	ux (test mode)	prose born false	DOCS FIXED this PR; real fix ha-9
S8	`postgres-requirements.md` said “no session-lifetime prepared statements” — pgx auto-prepares named statements per connection; the floor is PgBouncer ≥ 1.21 with `max_prepared_statements > 0` (PR-E’s negative control reproduced 305/480 statement failures at 0).	ops	docs lie (“works in CI, fails at the operator’s pooler”)	DOCS FIXED — floor stated, `default_query_exec_mode` escape hatch named
S9	“RDS Proxy multiplexes at transaction granularity” — AWS lists `SET`/`set_config` as PostgreSQL pinning triggers with no `SET LOCAL` carve-out; every `TxTenant` issues three. Safe (pinned = session semantics), multiplexing unverified.	ops	prose born false by generalization	DOCS FIXED — “safe; size the proxy for N × DB_MAX_OPEN_CONNS; verify on a live proxy before recommending”
S10	`docs/self-host.md` lede said multi-replica HA “is not in v1” while §Operating said it is the production posture.	ops	partial amendment	DOCS FIXED
S11	Verified-safe “login brute-force” entry and work-item 14’s template cited `lockout.go`/`FallbackFailureCounter`, deleted 2026-07-17 (#497).	ops	register rot	REGISTER FIXED (above)
S12	Hazard 3’s `migrate.go` comment clause was fixed by #395 the day the register was written.	ops	born-false clause	REGISTER FIXED (above)
S13	`hardcount_leader_lease_lost_total` reasons documented as `heartbeat`/`release`; the code emits `heartbeat_timeout`/`takeover`/`paused`/`released`, and a planned pause mid-tick would have paged.	ops	docs lie	DOCS FIXED (runbook + metric Help; `paused` excluded from the page)
S14	runbook-leader-leases said a deleted `leader_leases` row “never runs”; the ACQUIRE upsert recreates it.	ops	docs lie	DOCS FIXED
S15	MANUAL_TEST FLOW E7 still paused the email dispatcher with `pg_advisory_lock(76540004)`, which gates nothing since #870.	ops	stale flow	FIXED
S16	The Recreate-only upgrade for the release that crosses #870 lived only in ADR-114 §Rollout.	ops	missing deploy note	DOCS FIXED (CHANGELOG + deploy/README)
S17	`SHUTDOWN_DRAIN_DELAY` (default 5 s) had no sizing rule against the balancer’s health-check window.	ops	undocumented rule	DOCS FIXED
S18	Dashboard promises “Engine will retry: <time>” (`InvoiceAttention.tsx:96`) with no surface for a cluster-wide leader pause.	ux	missing readout	DEFER — trigger: first operator pause in production; fix = authenticated leader-status GET + one banner
S19	Dashboard invoice / credit-note creates send no `Idempotency-Key` (`api.ts:226/460`); subscriptions are covered by `UNIQUE(tenant, livemode, code)`. `apiRequest` never auto-retries, so this is the human re-submit-after-ambiguous-error class — identical at N=1.	ux	not multi-replica	DEFER to the idempotency backlog (#774 follow-up)

Contested, already decided: /health/ready is Redis-blind while the general limiter fails closed in production — recorded and decided in the SPOF register (“Tier: API state (Redis)”); no delta.

Refuted this sweep — do not re-flag: Reconciler.searchDisabled announce-once per process (benign, bounded by N per deploy); hardcount_scheduler_last_run_timestamp_seconds stamped by followers (“loop alive” was the pre-cutover contract; hazard 4 already amended); migrations through a transaction pooler (documented direct/session requirement); runbook alert thresholds × N series (each series is honest); SSE source comments (hazard 8 verbatim); the per-replica Stripe circuit breaker (N × 5 unknown outcomes, all reconciled — verified-safe stands).

Verified again on current main: sessions resolved from Postgres on every request; CSRF stateless; idempotency racing replicas DB-reserved; rate limiting one shared GCRA per Redis; every set_config is transaction-local; the only advisory locks left are transaction-scoped (credits, credit notes, bootstrap) plus the two documented session ones (bootstrap, migrations); exactly five leader-gated loops, five fenced funnels, and no HTTP / test-clock / operator path reaches a fenced funnel.