Sustained throughput: the run journal (2026-08)

Record, kept as written. These are the appendices of Sustained throughput, moved here on 2026-10-22. That page has the result, the method and what stops it.

Appendix A — the run journal

The chapters below are the runs as they happened, kept intact — including the readings that were later revised, because the revisions are part of the result. Each chapter states its per-run specifics; throughout, every repeat was gated and every event reconciled as stated in Method, so a chapter stands alone if linked directly.

The headline (2026-08-16, measured under the protocol below)

Rig: c7g.2xlarge app (8 vCPU Graviton, Hardcount in its own container) + c7g.xlarge load generator + db.m7g.2xlarge RDS PostgreSQL 16, all in ap-south-1a. ~$1.26/hr. Live-mode ingest path (rows with livemode=true — Hardcount’s live data, as opposed to test mode), 200 customers, one meter, each request a random customer. Table seeded to 22M rows before the first measurement and 47M by the last; 10 GB+ of indexes. Every figure below is open-loop, every repeat is gated (k6 thresholds held, 0 dropped, 0 failed, rows written == events claimed, Σ quantity == Σ gained, ≥1000 samples, no drift), and every event was reconciled against the database.

Rate-controlled — the numbers that are a service level

offered rate	batch	repeats	duration each	p50	p99	reads under load
200 ev/s	1	4/5 passed¹	10 min	4.1 ms (4.1–4.1)	4.9 ms (4.9–5.0)	responsive
1,000 ev/s	10	5/5 passed	10 min	7.2 ms (7.1–7.2)	8.2 ms (8.1–8.3)	responsive
5,000 ev/s	100	1/3, then a cliff	10 min	35–39 ms	51 ms → 116 ms → 9.9 s as the table passed ~60M rows	see § reads
10,000 ev/s	100	0/2 — a knee	10 min	38–40 ms	152 ms then 3.4 s	see § reads

¹ Repeat 1 failed its gate on a single 5-second stall 20 s in, coincident with autovacuum working through the 22M-row bulk seed finished minutes earlier (RDS write IOPS 2× baseline for the first five minutes); the other four were clean and the protocol now settles after seeding. Reported, not hidden.

5,000 ev/s is where this instance’s memory runs out. Repeat 1 held flat (p50 35 ms, p99 51 ms, first third 52 → last third 50). Repeat 2, thirty minutes and 6M rows later at 62M rows / 14 GB heap + 18 GB indexes on a 32 GB instance, rose from 50 ms to 308 ms in its last third. The evidence says exactly why: RDS CPU flat at 61–63 %, write IOPS flat, but read IOPS climbing 108 → 2,486/s through the run (the gp3 volume’s baseline is 3,000), read latency 0.6 → 2.6 ms and disk queue depth 0.5 → 10.7 in the final minute. The index working set had fallen out of cache and every insert was fetching index pages from EBS. Repeat 3, at 65M rows, delivered 4,035 of the 5,000 offered with a p99 of 9.9 s and 5,792 drops; the series was stopped there.

That is a capacity cliff, not a Hardcount defect, and it is the single most useful number here for a self-hoster: on db.m7g.2xlarge + 100 GB gp3, steady inserts turn read-IOPS-bound at roughly 60M rows / 30 GB of table + index. The levers are the usual ones — RAM (db.m7g.4xlarge = 64 GB), provisioned IOPS, fewer or smaller indexes (five on usage_events, one GIN), or time-partitioning so the hot working set stays small.

10,000 ev/s held for 90 seconds at p99 66 ms (rate ladder, below) — and did not hold for ten minutes. Repeat 1 delivered the rate with the tail climbing through the run (first third p99 70 ms, last third 195 ms; 4 drops), repeat 2 delivered 9,741 with a p99 of 3.4 s and 1,556 drops. RDS CPU went 68 → 73 % during the run and the table grew by 6M rows per repeat into 15 GB of indexes. That is a knee on db.m7g.2xlarge, not a plateau, and it is reported as one; this is what a 10-minute repeat is for.

Ceiling — closed loop, a maximum, never a service level

	reached	p50	p99
16 workers, batch 500, 90 s	25,424 ev/s (2,293,500 events, all reconciled)	296 ms	520 ms

The rate ladder (2 × 90 s per rung; the “held” column is the ingest gate)

batch	2,000	4,000	5,000	6,000	8,000	10,000	15,000
10	held (p99 10.6 ms)²	held (p99 31–36 ms)	—	not held (delivered ~5.5k, p50 1.5 s)	not held (~5.8k)	—	—
100	—	—	held (p99 42–44 ms)	—	—	held (p99 64–66 ms)	held (p99 262–315 ms, drift ×1.4)

² one of two repeats hit the same post-bulk-write stall as ¹; the other passed.

Second run, 2026-08-16/17: db.m7g.4xlarge, with the hot-row fix

Same app node and load generator; the database doubled to db.m7g.4xlarge (16 vCPU, 64 GB RAM, 100 GB gp3 — same 3,000 IOPS baseline). The three nodes cost ~$2.21/hr on-demand (ap-south-1 list prices: 1.916 + 0.196 + 0.098), which is ~$0.041 per million events at 15,000 ev/s. Hardcount ran at db7f86b0, which carries the fix for #818. Fresh 20M-row seed, 10-minute settle, then both ladders (2 × 90 s per rung), then sustained attempts. The first sustained attempt (10k) ran straight after the ladders on the table they left (61M rows); every later attempt started from a fresh 20M-row table (TRUNCATE, reseed, 10-minute settle), so a run is judged on its rate, not on the previous run’s leftovers. Table size at the start of each repeat is in the row.

The ladders — every rung, zero drops, every event reconciled

batch 10	p99	held?		batch 100	p99	held?
4,000	8.8 ms	yes		5,000	39.9 ms	yes
6,000	18.5 ms	yes — was the #818 wall		10,000	42.7 ms	yes
8,000	21.4 ms	yes		15,000	44.2 ms	yes
12,000	24.4 ms	yes		20,000	62 ms	yes (variance)
16,000	36–155 ms	yes (variance)		25,000	54.7 ms	yes
20,000	—	no (delivered ~18.3k, p50 0.9 s)		30,000	67.7 ms	yes
				40,000	—	no (delivered 30–36k, p50 1.7–2.3 s)

#818, verified on the rig: 6,000 ev/s at batch 10 went from “not held, ~5.5k delivered, p50 1.5 s, thousands of drops” to p50 7.2 / p99 18.5 ms, zero drops, and the batch-10 knee moved from ~570 requests/s to between 1,600 and 2,000. The 25,000 ev/s that was yesterday’s closed-loop ceiling is held open-loop at p99 55 ms.

The sustained attempts (5 × 10 min each)

offered	batch	passed	p50	p99 (passing repeats)	what stopped it
10,000	100	4/5	35.8 ms	41.6–49.3 ms	ran on the ladders’ table, 61M → 85M rows; repeats 1, 3, 5 flat (p99 41–49 ms), repeat 2 had a checkpoint bump early (queue depth 24, p99 100 ms, but no drift), repeat 4 failed on drift — its tail rose 42 → 67 ms through the run and 137–190 ms in the last 90 s with no write-IOPS, read-IOPS, CPU or memory signature during it and a 1,200-IOPS read burst the minute after; consistent with an insert-triggered autovacuum pass, unconfirmed. Repeat 5, at 85M rows, was clean; memory (FreeableMemory) never moved from 46.7 GB in the whole series
25,000	100	2/5	39.0 ms	52.6–65 ms	fresh 20M table; repeats 1–2 clean, then from repeat 3 on (51M rows +) checkpoint write bursts of 2,400–3,500 IOPS against the volume’s 3,000 baseline — disk queue depth 37–65 in the stall minutes, ReadIOPS ~0, RDS CPU 58–67 % — storage, not CPU or memory
12,000	10	4/5	8.1 ms	22.4–22.8 ms	fresh 20M table; four repeats within 0.4 ms of each other. Repeat 5 (49M rows) was clean for eight minutes, then its last two minutes ran p99 713 / 515 ms with 463 drops and p50 unchanged at 8.1 ms; the only concurrent signal is the run’s highest write IOPS (1,300 → 1,609, queue depth 1.5 → 3.2) — a checkpoint, but nowhere near saturation. 1,200 requests/s sustained on the recommended client shape — the hot row capped this at ~570
15,000	100	4/5	36.5 ms	43.3–44.6 ms	fresh 20M table; four repeats within 1.3 ms of each other, drift ×0.88–1.01, ReadIOPS ~0 throughout (the cache never missed — FreeableMemory flat at 46.7 GB); repeat 5, at 56M rows, hit one 50-second stall (23:52:00–23:52:50Z: disk queue depth 59, write IOPS 2,802 against the 3,000 baseline; 455 drops, p50 up to 1.9 s for that window). 49 of 50 minutes clean

Two of the four failed repeats have a clear storage signature over the whole minute — 25k and 15k: write IOPS at the volume’s 3,000 baseline, disk queue depth 59–65, p50 up to 1.9 s and hundreds of drops — checkpoint-era write bursts on a volume with no headroom. The other two (10k repeat 4, 12k repeat 5) are not attributed: re-reading CloudWatch for 10k repeat 4 without cutting the window at the run’s end shows its final minute at queue depth 28 and write latency 17–25 ms (a datapoint the first write-up missed), so it is not signature-free. But this rig had no 1-second instrumentation and pg_wal did not grow in either window, so the third run’s mechanism (below) is a candidate for them, not a finding.

Reading it as capacity: on this rig, steady batch-100 ingest above ~10k rows/s becomes bound by write IOPS during checkpoints — five indexes on usage_events and a 3,000-IOPS volume — before it is bound by CPU (≤67 %) or memory. At 15k one burst in fifty minutes crossed the line; at 25k they crossed every few minutes. The levers are provisioned IOPS / io2, max_wal_size and checkpoint spreading, or fewer indexes; none of them is Hardcount code, and none was pulled for this run — the numbers above are the stock RDS defaults on a 100 GB volume.

Ceiling and denominator on the 4xlarge (fresh 20M-row table, 10-minute settle)

	reached	p50	p99
closed loop, 16 workers, batch 500, 90 s	41,172 ev/s (3,713,000 events, all reconciled)	187 ms	226 ms

pgbench on the same database and row shape, batch 500, 16 clients, two interleaved A/B rounds (db-ceiling.sh), right after the ceiling run:

leg	round 1	round 2	median
A — plain batched INSERT, one transaction per 500 rows (the commit floor)	58,444 rows/s	46,982	52,713 rows/s
B — the same with Hardcount’s per-transaction RLS protocol (BEGIN, three set_config, INSERT, COMMIT)	45,195	36,154	40,674 rows/s

Hardcount’s closed loop, 41,172 ev/s, is 78 % of the raw commit floor and 101 % of leg B. At batch 500 the HTTP layer, auth, customer/meter resolve and the Go service add nothing measurable; the ceiling is the database’s own batched insert path with the RLS (row-level security) protocol on it (which costs 23 % of the floor here, 13–19 % on the 2xlarge at batch 1). Two things to keep in view: the rounds were 60 s each and each leg added ~3M rows, and both legs lost ~20 % from round 1 to round 2 as the table grew — the same growth sensitivity the sustained runs show, in a 4-minute window. So “101 %” is “within the round-to-round variance”, not a claim that Hardcount is free.

And the like-for-like cell for the run-1 comparison — batch 1, 16 clients, two interleaved rounds: A 7,184 rows/s (6,890 / 7,478), B 7,106 (6,943 / 7,269). At 16 clients a one-row commit is round-trip-bound (~2.2 ms per transaction, 16 in flight), so the RLS protocol costs ~1 % here against 13 % on the 2xlarge — the cost shows when the DB is CPU-bound, not when it is waiting on the client. (db-ceiling.sh also prints Hardcount-as-a-fraction lines; they are only meaningful when HARDCOUNT_EVS came from a closed-loop run at the same BATCH — the batch-1 legs above are not compared to the batch-500 ceiling.)

Third run, 2026-08-17: what a tail event is, caught live

Same shape (c7g.2xlarge app, c7g.xlarge generator, db.m7g.4xlarge, 100 GB gp3), Hardcount 97ef608b, this time instrumented for attribution: a parameter group with the logging on (log_autovacuum_min_duration=0, log_lock_waits with deadlock_timeout=200ms, log_min_duration_statement=250), Performance Insights, Enhanced Monitoring at 1 second, and a 5-second sampler on the app node reading pg_stat_io, pg_stat_wal, pg_stat_bgwriter, vacuum/analyze progress, the GIN pending list, page/extension locks and every backend’s wait event as one JSON line (scripts/bench-rig/db-sampler.sh; analysis: analyze-tail.py, tail-census.py; the Postgres log through pgBadger, a Postgres log analyzer). Control = stock RDS parameters, 12,000 ev/s at batch 10, 5 × 10 min from a fresh 20M-row table. Postgres 16.14; RDS defaults that matter here: wal_segment_size 64 MB, max_wal_size 6144 MB on this class, min_wal_size 192 MB, wal_keep_size 2048 MB, checkpoint_timeout 300 s, checkpoint_completion_target 0.9, archive_timeout 300 s, wal_init_zero on.

(The WAL mechanism, why the pool runs dry, and the round-trip cost model that this chapter originally carried inline are in Appendix C.)

The control caught one

Repeat 1 held (p99 24.5 ms, 0 drops) but carried a 45-second event at 04:52:06–04:52:49Z: 10-second p99 120–177 ms against a 22 ms baseline (1-second p99 up to 230, worst request 279 ms), p50 untouched. Inside it, from the raw k6 samples: exactly ten stall-seconds, 4–5 s apart, each 0.17–0.28 s long (14–24 % of the requests in that second slowed, 3.5 % over the whole window). What every source said about those seconds:

source	what it showed
Enhanced Monitoring, 1 s	the rdsdev device at 10,395 write IOs/s of ~13 KB = 133 MB/s, util 95 %, device-mapper queue depth 40–96 — the volume at its throughput ceiling — while the NVMe beneath took them merged (~1,600 IOs/s of ~80 KB, queue 5, await 3 ms; read live from the EM stream, now retained by the fetch step). Roughly 100 MB of writes above baseline in each stall second, ~1 GB over the window
the same minute in CloudWatch	1,039 IOPS, 42 MB/s, queue depth 1.3 — a 60-second average
Postgres’s own writers (pg_stat_io, 5 s)	unchanged: checkpointer a perfectly even ~1,290 buffers/s (10 MB/s) through the whole checkpoint, client backends 0 writes / 0 evictions (only relation extends, ~930/s), WAL 14 MB/s, wal_buffers_full = 0
sampler tick 04:52:11	one client backend in IO:WALInitSync, 19 in LWLock:WALWrite
Performance Insights, 1 s	LWLock:WALWrite = 18 sessions at 04:52:15 (PI’s 1-second samples otherwise mostly miss ~0.2 s stalls)
Postgres log	the checkpoint before the event (04:48:53, the first after warm-up) recycled 29 and unlinked 10 old segments; the one completing at 04:52:48 recycled 50
CloudWatch TransactionLogsDiskUsage	pg_wal 6.44 → 5.77 GB at 04:49 (10 fewer files), back to 6.44 GB during the event
not implicated	autovacuum (its passes fell outside the window; autoanalyze 0.5 s each), the GIN pending list (5–486 pages, its normal sawtooth), buffer eviction (0), lock waits (0 lines)

A committing backend creating a WAL segment. When the next 64 MB segment is needed and no pre-made one is waiting, the backend writes 64 MB of zeros (256 KB pwritev calls) and fsyncs the file, holding WALWriteLock throughout — every other commit waits (IO:WALInitWrite/WALInitSync, xlog.c XLogFileInitInternal, reached from XLogFlush). At 14 MB/s of WAL that is one segment every ~4.7 s, each ~0.2–0.28 s here; segment size sets the stall’s grain, the flush rate its length. Ten stalls = the ten segments the 04:48:53 checkpoint had not recycled.

Provoked on demand: idle 15 min, then 12k ev/s for 5 min

arm	min_wal_size	max_wal_size	10-s buckets > 5×	worst 10-s p99	seconds at cap	what pg_wal / the pool did
A — stock	192 MB	6 GB	3	254 ms	6	flat 6.44 GB through the idle (nothing removed); stall began 07:34:18, ~300 s after load started, after ~63 segments of WAL — the pool at resume — with the timer checkpoint (started 07:32:23) still writing; +0.47 GB (7 segments created) by the time the 5-min run ended
B — floor only	6 GB	6 GB	5	196 ms	7	same shape: dry after ~57 segments, +0.80 GB created before the run ended
C — depth only	192 MB	16 GB	3	162 ms	3	−1.21 GB during the idle (18 unlinked, path 1) then a stall at 09:01:30–09:01:56
D — floor and depth, applied to a shallow pool	16 GB	16 GB	13	234 ms	20	pg_wal 5.10 → 6.98 GB during the run: the setting does not create segments — every new one was built under load, so the whole run stalled every ~5 s (the transition cost of raising the setting; pg_switch_wal() to pre-grow is not available to rds_superuser)
D2 — the same, after the pool had grown to depth (a 12k series ran in between)	16 GB	16 GB	0	24.7 ms	0	flat at 12.15 GB through idle and resume; no WALInit*, no pile-up — the same recipe that stalled A–D

The 50-minute series with min_wal_size = max_wal_size = 6 GB (T3, same load as the control) had no sustained event (>5× buckets 5 → 0; in-run seconds at the cap with small IOs 12 → 0; WALInit* sightings 1 → 0; pg_wal flat). But it is a null test of the floor: its distance estimate never decayed below the point where the floor would bind, so the treatment was never exercised. The clean result is down to checkpoint phase, and it is one series. Arm B is the honest verdict on that setting.

The setting under the full series (T5): 12,000 ev/s × batch 10, 5 × 10 min

Same load and protocol as the control, min_wal_size = max_wal_size = 16 GB, pool at 147 pre-made segments when the series began (the reseed had grown it):

	control (stock)	min_wal_size = max_wal_size = 6 GB (T3, null test)	min_wal_size = max_wal_size = 16 GB (T5)
repeats passed, p99 range	5/5, 21.8–24.5 ms	5/5, 22.1–23.7 ms	5/5, 22.4–23.3 ms
10-s buckets > 3× / > 5× the series median	5 / 5	2 / 0	0 / 0
worst 10-s p99	177 ms	75 ms	52 ms
seconds at the throughput ceiling with small IOs, inside repeats	12	0	1
sampler ticks with a backend in WALInit*	1	0	0
checkpoints timed / WAL-driven	3 / 10	2 / 9	10 / 0
pre-made segments, min / median (pg_ls_waldir)	not sampled	not sampled	45 / 99
segments unlinked at checkpoints	10	0	0

With the pool deep, checkpoints are time-driven (so the recycling formula’s 10 % slop is back in play), the pool never fell below 45 segments, and no tail window appeared. One series each — but three of the four provocation arms stall on demand and this setting is the only one under which the pool never got near empty. And the provocation that stalled every other arm (D2 in the table above) did not stall it.

Then the same setting at 15,000 ev/s × batch 100 (T6, 5 × 10 min, fresh 20M): 5/5, p50 33.6–34.0 ms, p99 40.2–42.8 ms, 0 drops, no tail bucket, worst 10-s p99 88 ms, all 9 checkpoints time-driven, WAL 17.6 MB/s, pool minimum 30 pre-made segments (median 82) — thinner than at 12k, as the rule predicts when the rate rises. Last night’s stock run at this rate was 4/5 (43.3–44.6 ms) with a 50-second volume-saturation stall. Caveat on repeat 5: the operator’s laptop slept mid-series; k6 ran to completion on the load generator (90,001 iterations, 9,000,100 events, matching the row count) and its summary and samples were read afterwards. Its p50/p99/drift/drops therefore come from the same files as every other repeat, read by hand.

And at 25,000 ev/s × batch 100 (T7, same protocol): 4/5, p50 35.6–36.1 ms, p99 49–120 ms, 0 drops, every event reconciled — the one failure a drift gate (repeat 2’s last third ×2.9 its first, p99 67 ms), not a stall. Compare stock last night: 2/5, p50 up to 1.9 s, hundreds of drops, disk queue depth 37–65. Here the pool is not deep enough and the evidence says so: WAL runs at 29.6 MB/s, so the rule asks for ~19 GB and 16 GB is short. The sampler shows the pre-made pool at 0 segments at 17:10 and 17:25, two checkpoints turned WAL-driven (starting: wal), 471 seconds at the throughput ceiling with small IOs, two WALInit* sightings, 18 buckets between 3× and 5× (none above; worst 10-s p99 208 ms), and total WAL files growing 181 → 244 through the series — the created segments deepening the pool, which is why repeats 3–5 (p99 49–64 ms) are cleaner than 1–2. So the boundary is measured: 16 GB covers 12–15k on this rig; at 25k size to ~24–32 GB (or shorten checkpoint_timeout).

The same reading offers a candidate for last night’s stock 25k/15k stalls that this write-up still classifies as storage-bound: at 29.6 MB/s a stock pool of ~50 segments is consumed in ~108 s — dry at every cycle end — and the segments created are unlinked at the next checkpoint, so TransactionLogsDiskUsage stays flat while the churn’s 8 KB writes consume the volume’s 3,000 IOPS. It fits the 2,400–3,500 IOPS / queue 59–65 signature and the fact that the same load with the pool sized showed no saturation and no drops; it is not provable from that rig’s data, so it stays a candidate.

Re-reading run 2 (2026-08-19)

Run 2’s two unattributed events, re-read at 1 second from their raw k6 samples (2026-08-19): they have different fingerprints. 10k repeat 4 (79M rows): in its last 30 s, whole seconds in which every request took +50–100 ms (p50 70–140 ms), recurring every 5–9 s, each ~0.15–0.25 s — the segment-creation rhythm, visible in p50 only because a batch-100 request takes ~36 ms; with the pool at its cap the created files are unlinked at the next checkpoint, which is why pg_wal read flat at 1-minute resolution. Most likely this mechanism; not proven (no 1-second DB view that night). A later instrumented 10-repeat series at 10k b100 on stock settings (2026-08-19, 30M → 90M rows, covering E2’s band) did not reproduce E2 in its band, but showed the mechanism itself twice at 36M rows: the pool at zero, files created, and a ~150–180 ms freeze hitting 12–16 % of that second’s requests at every creation, p50 flat — the same signature at a second load shape. 12k repeat 5 (49–56M rows) (E1): something else — 1–3 s global freezes (p50 750 ms, 100 % of requests slow, all 463 drops in the first one) recurring every ~30 s for two minutes (22:31:40, 22:32:12, 22:32:43, 22:32:49). Not the 5-second rhythm. Attributed on 2026-08-19, from two instrumented instances of the same shape on a stock-settings 25k series (peer-session rig, vacuum log on; verified from the raw files by both sessions): the end of an insert-triggered autovacuum pass on usage_events. The cleaner instance: sampler ticks show the worker at “scanning heap” 2,554,157 of 2,587,098 pages (19:22:01), then “cleaning up indexes” on IO:DataFileRead (19:22:06), and the log’s automatic vacuum of table … usage_events completing at 19:22:09 — the second the event ends. In those seconds Enhanced Monitoring shows the volume taking 17,700–21,100 IOs/s of ~6 KB, disk queue depth 348 and 529, await 20–25 ms (normal seconds: 700–1,600 IOs/s of ~107 KB, queue 1–3 — the IO-size collapse is the signature that tells a vacuum storm from a checkpoint burst at a glance); Performance Insights shows 5–14 sessions on LWLock:WALWrite; p50 lifts 36 → 64 ms with 96–99 % of requests affected for six seconds; the WAL pool stays at 17–20 files (not the segment mechanism). The vacuum’s own report says what the writes were: 544,046 pages scanned, 540,288 dirtied, 6.6M tuples frozen, average write rate 106.7 MB/s — the vacuum rewrites nearly every page added since the previous pass (hint bits and opportunistic freezing, which data_checksums=on pushes through full-page writes), and RDS’s autovacuum_vacuum_cost_limit on this class (1,200, 2 ms delay) lets it write at the volume’s full throughput; every committer’s WAL write queues behind it. The second instance (19:07:55, one second after its logged completion at 19:07:54; queue depth 143) is consistent-with rather than clean — a one-second alignment the 1-second EM cannot resolve. Run 2’s E1 fits (global freezes, +300 IOPS in the minute average, pg_wal flat, 49–56M rows = where a ~20 %-growth vacuum lands in a 12k series) and is written as most likely this, since that rig kept no vacuum log. Levers, untested, stated as an open question: pace the vacuum (autovacuum_vacuum_cost_delay up / cost_limit down) so it writes under the volume’s headroom — shallower but longer storms, which may be worse for a p99 budget — or vacuum smaller slices more often (autovacuum_vacuum_insert_scale_factor below the 0.2 default) so each rewrite burst is smaller. One arm decides which. The 15k/25k failures: the stock-25k collapse did not replicate. A 2026-08-19 stock-settings 25k b100 series (fresh 20M, 5 × 10 min, 20M → 81M rows) ran 0 drops throughout with no minute-scale saturation at any size; its run 1 carried a 32-second pool-rhythm episode (29 of 32 freeze-seconds with the pool at zero, total files 96 → 102) and the pool never ran dry again — at 25k the stock pool churns as a start-up transient and stops once grown to a working depth. So the run-2 archive’s repeats 3–5 (64 / 145 / 172 freeze-seconds, p50 in seconds, queue depth 37–65) need a separate explanation the archive cannot supply; the pool-churn candidate is weakened for them, and they stay “storage-bound, not replicated”.

What this measured is a steady, well-behaved client

The rate is constant and evenly spaced, keys are never retried, timestamps are server-side, batches are uniform, one tenant, one meter. That is the right first benchmark and what every peer publishes; it is not a burst test and it does not claim to be. A spike scenario and a retry mix are the two cheapest next steps and are not part of this run.

---

What a laptop can and cannot tell you — revised

This benchmark existed on a laptop first, and an earlier version of this page called the laptop 9× optimistic because it reported 1,800 ev/s at p99 ≤ 50 ms where the first AWS run held 200. That verdict was wrong, and it is worth saying why: the first AWS run’s latencies came from a load generator with a lockstep-burst bug (its workers fired in step, bunching requests into bursts; since deleted), on the test-mode path, against a one-customer fixture. Under the current protocol the same class of hardware holds 10,000 ev/s at p99 66 ms through the full HTTP path — the laptop’s in-process figure was, if anything, pessimistic by ~5×.

What did transfer from the laptop, and still does:

laptop finding	held on the rig?
COMMIT dominates in-database time; batching is the lever	Yes — the batch ladder is the whole story above
the batch curve flattens after ~50	Yes in shape
~26 statements per single-event request	Yes — measured on both
the per-request wall (hot row)	Not visible on a laptop — needs a real RTT to bite at a rate a laptop can offer

Use a laptop for ratios and mechanisms; use the rig for absolutes and for anything that depends on a real round trip. And never let the generator’s bugs be attributed to either.

Appendix B — superseded figures

Superseded: the 2026-08-15 measurements

For the record — the figures this page carried before the protocol existed, each a single run from a since-deleted generator with a lockstep-burst bug, on the test-mode path, against a one-customer fixture:

configuration	published then	under the protocol now
200 ev/s single-event	p99 119 ms	p99 4.9 ms (4/5 × 10 min)
1,000 ev/s batch 10	p99 269 ms	p99 8.2 ms (5/5 × 10 min)
closed-loop, batch 500, db.m7g.2xlarge	10,203 ev/s	25,424 ev/s
“$0.032 per million events”	compute-only, at that ceiling	~$0.014/M at the new ceiling, same basis; ~$0.035/M at 10,000 ev/s open-loop (app + DB compute, on-demand, ~$1.26/hr)

The mechanisms the earlier page identified — commits per event as the lever, batching as the requirement, same-AZ as the placement decision — were right. The absolutes were not, for reasons that were entirely the measurement’s.

Appendix C — mechanisms

The mechanism from first principles (read this if the rest is Greek)

1. Every commit is written to the write-ahead log (WAL) first, and the commit is not acknowledged until that write is on disk. That is what makes a billing event durable.
2. The WAL is a sequence of fixed-size files (“segments”; 64 MB on RDS). Filling one and moving to the next happens ~every 4–5 s at 12,000 events/s.
3. Postgres keeps a small stock of ready-made, already-formatted segments ahead of the current one, so switching is instant. The stock is refilled by recycling at each checkpoint (~every 5 min): old segments that are no longer needed for crash recovery are renamed into future ones.
4. If the stock is empty when the next segment is needed, the transaction that needs it must create the file — write 64 MB of zeros and flush it — while holding the lock every other commit needs. Everyone freezes for the ~0.2–0.3 s that takes. Until the next checkpoint refills the stock, this repeats at every segment boundary: a freeze every ~5 s. Typical writes (p50) are unaffected; the slowest 1 % (p99) jump 5–10×.
5. The stock is sized to about one checkpoint’s worth with almost no slack, by Postgres’s recycling rule at these settings. Two ordinary events remove the slack: a quiet period makes checkpoints small, which makes the next one recycle fewer files (RDS’s min_wal_size = 192 MB is the floor that would prevent it), and after an idle RDS’s 2 GB of retained log sits inside the window, so the stock on resume is shallower than usual and the first timer-driven checkpoint refills it too late (max_wal_size bounds the depth). Both were reproduced on demand.
6. The fix is a deeper stock: min_wal_size = max_wal_size big enough for the worst refill gap at your write rate (wal_keep_size + 1.9 × WAL rate × checkpoint_timeout; 16 GB here). The stock is a revolving buffer — used and refilled every cycle, never “consumed” — so it costs disk space and nothing else. It grows only as segments get created, so set it at provisioning (a bulk load grows it for free); raised on a live instance it fills over ~15 minutes of peak traffic, stalling in the meantime, then stops.
7. What it does not fix: if the volume cannot absorb the write rate (15–25k events/s on a 100 GB gp3), that is capacity, and only a bigger volume or less WAL per event helps.

Why the pool runs dry — from the source

Pre-made segments come from recycling at checkpoint completion: XLOGfileslop keeps segments up to 1.1 × (1 + checkpoint_completion_target) × distance_estimate beyond the redo point, clamped to [min_wal_size, max_wal_size]; a WAL-driven checkpoint fires at max_wal_size / (1 + target) (CalculateCheckpointSegments). At this write rate the estimate converges to that distance, 1.1 × 1.9 × D exceeds max_wal_size and the clamp binds — so the future pool at each completion is max_wal_size − 0.9·D ≈ 50 segments, and the next completion lands 49.4–50 segments later. The margin is under one segment, not the formula’s 10 % (that slop only exists once 1.1 × 1.9 × D < max_wal_size, i.e. when checkpoints are time-driven). Two things then take the margin away, both after a quiet spell:

1. Fewer segments recycled. Small checkpoints decay the distance estimate (10 % each: 50.0 → 48.5 → 43.9 → 40.2 segments across the three that bracketed the control’s warm-up); once 1.1 × 1.9 × estimate falls under the clamp the recycle target drops and old segments are unlinked instead of recycled (“10 removed”). min_wal_size is the floor that would stop that — RDS’s 192 MB never binds. The next cycle is those segments short.
2. Retained segments inside the window. RDS keeps 32 segments (2 GB, wal_keep_size) behind the end of WAL. In steady state those sit before the redo point and cost nothing; after an idle (redo ≈ end of WAL) they sit inside the max_wal_size window, so the pool on resume is ~63 segments, not ~95 — and the first checkpoint after resume fires on the 5-minute timer and only refills when it completes ~270 s later. Neither min_wal_size nor max_wal_size alone changes that arithmetic.

Why: round trips, and where the time goes

The RDS round trip in the same availability zone is ~1 ms. On the production path a single-event request costs ~26 database statements (auth, customer resolve, meter resolve and the insert each open a transaction of their own — BEGIN, three set_config calls for row-level security, the query, COMMIT — plus the last_used_at touch), measured with log_statement=all against an idle control. Batching amortises everything except the per-request part: ~3.4 statements per event at batch 10, and the request rate — not the event rate — is what the hot row above caps.

pgbench on the same database and row shape (db-ceiling.sh) puts the database’s own one-row commit floor at 6,840/s (16 clients) and 14,284/s (64), and the same with Hardcount’s per-transaction RLS protocol at 5,961 and 11,546. So the three set_config round trips cost 13–19 % at the DB, and Hardcount’s 25k ev/s ceiling is a fraction of what the DB can absorb — the remainder is HTTP + auth + resolve + the Go service, and, at high request rates, the row.