Release checklist: live services

Run this by hand for each release. It covers what automated tests cannot prove, because they fake the provider: Stripe, email, LiteLLM and the invoice PDF. Use Stripe test mode only. See How we test.

When to run each part:

Part	When
Setup to Email, PDF, LiteLLM	Before you tag the release
Fresh install and upgrade and First invoice tutorial	After the tag’s image build has pushed the release images, and before you publish the release notes

The checks write Stripe-linked test data into the database the server uses. Point the server at a fresh or dedicated database if you want to keep yours clean.

Setup

• cp .env.example .env, then make up (Postgres, Redis, Mailpit), make bootstrap (prints the sign-in and the test keys), make dev (API on :8080) and cd web-v2 && npm run dev (dashboard on :5173).

• The .env.example defaults send email to Mailpit and point email links at the dashboard on :5173. Read mail at http://localhost:8025.

• Settings → Payments: paste the Stripe test keys. Run stripe listen --forward-to localhost:8080/v1/webhooks/stripe/<hc_spc_id> and paste its whsec_… back.

• Stripe test cards:

  Card	Result
  4242 4242 4242 4242	Succeeds
  4000 0000 0000 0341	Saves, then declines on charge
  4000 0027 6000 3184	Asks for 3-D Secure

• In Checkout, choose “Pay without Link” if offered, and give a ZIP for a US address, or Save does nothing.

• Stripe end-to-end test has the step-by-step commands for the Stripe checks.

Stripe

• Wrong keys are refused: a made-up sk_test_ key shows “Stripe didn’t accept these keys”.
• Charge: the first invoice of a subscription whose base fee bills in advance is charged to 4242, Hardcount shows it paid, and the Stripe dashboard shows the same amount.
• Webhook: stripe listen shows payment_intent.succeeded answered with 200.
• Hosted invoice page: Pay with 4242 → “Paid” banner within seconds, and the card appears on the customer.
• Hosted invoice currency: first make sure the test can fail. Use a US test account with Adaptive Pricing on (Dashboard → Settings → Adaptive Pricing), and check that a plain session for a +location_IN email offers INR: stripe checkout sessions create -d mode=payment -d success_url=https://example.com -d customer_email=test+location_IN@example.com -d "line_items[0][quantity]=1" -d "line_items[0][price_data][currency]=usd" -d "line_items[0][price_data][unit_amount]=1000" -d "line_items[0][price_data][product_data][name]=control" → open its url. If INR is not offered, this step proves nothing on this account. Then, for a Hardcount customer whose email has +location_IN, open the hosted invoice page and Pay → the session has adaptive_pricing.enabled=false and Checkout shows only the invoice’s currency. After paying with 4242, the PaymentIntent shows the invoice’s amount and currency and has no presentment_details.
• Hosted invoice Pay on an Indian account: on a test account registered in India (Stripe doesn’t offer Adaptive Pricing there), Pay on a hosted invoice → Checkout opens. Hardcount sends adaptive_pricing[enabled]=false on every hosted Pay session, so a Stripe error here breaks Pay for every invoice of such a tenant.
• 3-D Secure: Pay with 4000 0027 6000 3184, fail the challenge → the invoice stays unpaid; pass it → paid.
• 3-D Secure off-session: save 4000 0027 6000 3184 through Add payment method, then finalize a one-off invoice → Stripe answers authentication_required; the notice reads “Customer must confirm payment” with only Email payment link; Mailpit has “Action required — confirm your payment for invoice …” with no word of a decline. Pay on the hosted page and pass the challenge → paid.
• Refund: a credit note with “Refund to card” → refund_status=succeeded, and the refund shows in the Stripe dashboard.
• Key in use: start two stripe payment_intents create -d amount=2000 -d currency=usd -d payment_method=pm_card_visa -d "payment_method_types[]=card" -d confirm=true -i <key> at once with one new key. Repeat with a fresh key until one answers HTTP 409 with code idempotency_key_in_use (identical replies are a replay) → write its error type in the comment on stripe409KeyInUse in internal/payment/stripe_409_test.go. Hardcount treats every 409 as unclear, whatever the type (#976).
• List after create: create a PaymentIntent for a test customer with metadata hardcount_invoice_id=inv_x, then at once stripe payment_intents list --customer <cus> -d "expand[]=data.latest_charge" → it is listed, with latest_charge.refunded. The stale-hold sweep reads “not listed” as “Stripe never got the charge” (ADR-127).
• List for a customer this account does not have: stripe payment_intents list --customer cus_doesnotexist → write down whether Stripe answers an empty list or a resource_missing error. On an error the sweep keeps the hold and logs “can’t confirm an earlier charge with Stripe”; on an empty list it clears it.
• Interrupted charge: hold Stripe’s response in a TCP proxy (start the API with HTTPS_PROXY pointing at it), start Collect payment on a test invoice, and SIGKILL Hardcount once Stripe has the request, within the client’s 30-second timeout. After restart the invoice refuses Collect payment with “being confirmed with Stripe”; the first reconciler run after the hold is 10 minutes old marks it paid (about 18 minutes after the kill in local mode), with one PaymentIntent in the Stripe dashboard.
• India: on a test account registered in India, the same list call works (Search is not offered there).
• Stripe Tax: with a US state registration in the Stripe test account, tax provider “Stripe Tax”, customer with a full US address → the invoice has a tax line that matches Stripe’s calculation.

Money walk across a renewal

A dunning run is the automatic retries after a failed payment.

• Dunning policies page: “Default retries” is the default and enabled. A bootstrap creates it in both modes (ADR-123). Its row reads “3 retries, on day 3, 7 and 14”. When its retries run out it changes nothing: final actions none and none. Running the demo first does not change this: a recipe creates no dunning policy.
• Mint a test clock, create a customer with an email, pinned to the clock, and subscribe it to a plan with a base fee (the dashboard’s plan form starts on in arrears; the API needs base_bill_timing).
• With stripe listen running, open the customer’s setup link (Add payment method) and save card 0341 in Checkout → the card is listed on the customer. Do this before you advance the clock.
• Advance the clock past the renewal → the renewal invoice is declined and a dunning run starts.
• Open the invoice link in the payment-failed email and pay with 4242 (Checkout offers the saved 0341 first: click Change) → the invoice is paid, and the dunning run shows recovered.
• Price change on a test clock: give the plan a usage price and send usage. Change the price on the plan’s page → before you save, the page says the change applies from each subscriber’s next billing period. Advance the clock past the renewal → that invoice bills the old price; send usage and advance again → the next invoice bills the new one. Its PDF usage line names the meter and the price’s label values, with the new rate.
• A new subscriber after a price change, on a test clock: change a usage price of a plan that has a subscriber, advance the clock an hour, then subscribe a new customer of the clock to the plan and send usage. Advance past its first renewal → its first invoice bills the new price.

Email, PDF, LiteLLM

• Settings → Email → Send test email: the page says “Sent to” your address and the message arrives in Mailpit. With a wrong SMTP_PORT, the page shows the mail server’s error instead.

• Invoice → Email: the message arrives in Mailpit; open it, read it, and check the amount and the links.

• The emailed PDF opens, and its total, lines and tax match the invoice page. Its buyer address matches the hosted invoice page (the email’s link); the dashboard invoice page shows no address.

• Run a real LiteLLM proxy with the callback from the LiteLLM guide → one completion creates one input and one output usage event on the customer (plus cache_read if the prompt was cached).

• With the same proxy, send one completion on a model the customer’s plan has no price for → hardcount_usage_unbilled_total{reason="no_price"} on /metrics goes up by the call’s events, and the customer page shows them in its “not billed” row.

• Open Anthropic’s pricing page and the models overview. Update anthropicListPrices and anthropicPricesChecked in internal/recipe/anthropic_prices_test.go. If a price moved, fix anthropic_style.yaml, bump its version, and add a CHANGELOG line telling existing installs to update that price. If a model was added, also:

  • Add its id to modelFamilies in internal/integrations/litellm/mapper.go. If its price depends on the prompt’s length, add it to promptTiers too.

  • Add its prices to the plan in anthropic_style.yaml, bump the version, and add a CHANGELOG line telling existing installs to add its prices.

  • In internal/integrations/litellm/model_families_test.go, add it to pricingPageClaudeIDs, and remove it from mustStayRaw if it is there.

  • In internal/integrations/litellm/testdata/list_claude_model_ids.py, add it to FAMILIES and PAGE, and remove it from PROBES if it is there. Then regenerate the list (do not edit it by hand):

    pip download litellm==1.83.9 --no-deps -d /tmp/litellm-1.83.9
    unzip -o /tmp/litellm-1.83.9/litellm-1.83.9-py3-none-any.whl \
      litellm/model_prices_and_context_window_backup.json -d /tmp/litellm-1.83.9
    python3 -I internal/integrations/litellm/testdata/list_claude_model_ids.py \
      /tmp/litellm-1.83.9/litellm/model_prices_and_context_window_backup.json \
      > internal/integrations/litellm/testdata/litellm-1.83.9-claude-model-ids.tsv

  Then run go test ./internal/recipe/ ./internal/integrations/litellm/.

Fresh install and upgrade

These prove that the images the quick start pulls work, and that an existing install can move to them. Use a machine that has Docker but no Hardcount containers or volumes, for example a new VM.

Fresh install

• Clone the release tag, then run ./quickstart.sh in deploy/compose. The output names the new release (“Starting Hardcount v0.X.Y from the published images.”) and ends with a login and a test key.
• docker compose exec hardcount-api /usr/local/bin/hardcount version prints the new release.
• ./quickstart.sh demo ends with links to a finalized invoice and its customer, and both open in the dashboard.

Upgrade drill

Run it from v0.9.1 on. It was skipped for v0.9.0, the first release named Hardcount: its previous release, Velox v0.8.2, has no upgrade path. The .env that v0.8.2’s quickstart writes names VELOX_* settings, so step 2 stops at docker compose pull with required variable HARDCOUNT_APP_DB_PASSWORD is missing a value. With the settings renamed by hand, hardcount migrate status says no migration record for a v0.8.2 database, and made by Velox for one a build of Velox’s main made after the switch to goose (#1198); start-up refuses both.

1. Run ./quickstart.sh destroy to start from an empty install. Check out the previous release tag and run ./quickstart.sh, then ./quickstart.sh demo. You see: the demo’s invoice link. Open it and note the invoice number, total and line amounts. Download its PDF.
2. Check out the new release tag. In deploy/compose/.env, change the tag on the HARDCOUNT_IMAGE and HARDCOUNT_DASHBOARD_IMAGE lines to the new release. Run docker compose pull, then docker compose run --rm hardcount-api migrate status. You see: the plan, with running now: no. The state is behind, listing the release’s new migrations.
3. Run docker compose up -d --no-build. You see: hardcount-api is recreated, and hardcount version prints the new release.
4. Check the migrations: docker compose logs hardcount-api | grep migration. You see: one migration applied line per migration, then migrations applied, with version set to the new release’s last migration number, and no error. hardcount migrate status now says up to date.
5. Open the invoice from step 1. You see: the same number, total and line amounts. Its PDF has the same lines and amounts as the one you downloaded. The “Generated on” time at the bottom differs, because each PDF is made when you open it. A release that changes what the PDF prints says so in its notes.
6. Open the Dunning policies page. You see: the same policies as before the upgrade. An upgrade never adds or changes a retry policy on an existing account.
7. Run ./quickstart.sh demo again. You see: a second finalized invoice for a new demo customer.
8. Run ./quickstart.sh destroy when you are done.

First invoice tutorial

• Run Send usage and get your first invoice end to end against the quick-start stack from the fresh install. Every command works as written, and each output matches the one under its “You see:”. CI runs the same commands on every pull request, against a build from source, not the release images. When a step differs, fix the page on main before you publish the release notes. The page links its own version for the release that quickstart.sh starts. When you change HARDCOUNT_VERSION there, change the page’s link and version too: TestFirstInvoicePageNamesTheQuickStartRelease fails until you do.