Shipped at five, broken by eight: rolling back a bad release

19:52: checkout errors

Release 4.12 of the billing portal goes out at 17:05 with a new invoice PDF and an upgraded payment SDK. The smoke test passes and everyone goes home. At 19:52 a Grafana alert lands in Caliptic: checkout errors have jumped to 8.4 %. Event Management raises an incident, and the Triage Agent marks it urgent — customers can't pay, though invoices and login still work.

What changed?

The first question on any outage is what changed, and the change record answers it: release 4.12 reached billing-portal-01 at 17:05, the only change on that server in a week. Two customers have already written to support; their tickets are linked to the incident, and Omar, on call, is paged.

The cause, from the logs

The DevOps agent reads the logs: every card payment fails with an invalid token. The new SDK refreshes its token against the payment provider's v3 endpoint, but the merchant account is still on v2. The token fetched at deploy time lasted about two and a half hours — which is exactly why the release looked healthy until the evening.

Roll back or fix forward?

• The agent lays out both options: roll back to 4.11 in about ten minutes, or fix forward with a change nobody has tested. It recommends the rollback.
• Omar decides, on the record: roll back now, fix forward tomorrow in daylight.
• The rollback is an emergency change. The agent drafts the steps from the original change's rollback plan; one approval is enough.
• At 20:29 payments work again. 37 payments failed in between, and the service desk sends each of those customers a retry link.

The next morning

Release 4.12 is marked as rolled back, and the cause becomes a known error with a clear workaround: stay on 4.11. In the morning the agent prepares 4.12.1 — one line of configuration and a regression test that runs past the token's lifetime. The change board approves it for 11:00, with the whole team watching, and the first token refresh that afternoon goes through.

Closing the loop

• The review of 4.12 is honest: the tests never ran longer than a token lives, so the failure surfaced after everyone had gone home.
• An improvement goes on the board: a synthetic test payment every five minutes, and no payment-code releases after 16:00.
• The rollback becomes a runbook in the knowledge base.
• The Ops Assistant writes the management summary straight from the records.

What made the difference

• Changes and incidents live in one place, so "what changed?" took seconds.
• An agent did the log reading and laid out the options; a person made the call.
• Rolling back was a planned path, not an improvisation: the rollback plan was written before the release.
• The fix waited for daylight, and came with the test that would have caught the problem.