The log that ate the disk: a PostgreSQL emergency at 3 a.m.

02:47: disk at 96 %

Alertmanager wakes Caliptic: billing-db-01 is at 96 % disk, and at the rate tonight's invoice run is writing it will be full in about 45 minutes. When the disk fills, PostgreSQL stops, and billing stops with it. There had been a warning — 85 %, the morning before — but it had no owner, so nobody looked.

Where the space went

Event Management raises an urgent incident. The Triage Agent links the earlier warning, pages Omar, who is on call, and hands the investigation to the DevOps agent. The agent finds the space in one folder: pg_wal, the write-ahead log, at 131 GB where it is normally about 2. PostgreSQL keeps every WAL segment until it has been archived to backup storage, and archiving has been failing for four days with access denied.

What changed four days ago?

A key rotation for the backup storage. The change record listed the billing portal and the backup job as consumers of the key — but not the PostgreSQL WAL archiver, because the CMDB showed no relation between the database and the backup storage. The archiver kept the old key, and every attempt failed quietly.

Two steps and one rule

• Extend the data volume by 100 GB, online, to buy time.
• Give the archiver the new key and reload PostgreSQL, so the backlog drains on its own.
• And the rule: nobody deletes files in pg_wal by hand. It can corrupt the database and breaks point-in-time recovery.
• Omar agrees on the record. The work runs as an emergency change with one approval.

The rest of the night

By 03:25 the disk is at 65 % and falling, 38 minutes after the alert. Billing never stopped and no customer noticed. By 05:10 the archiver has caught up: the write-ahead log is back to 2.1 GB, the disk to 22 %, and point-in-time recovery is verified against the archive.

The morning after

• The problem record names all three gaps: the rotation missed a consumer, failed archiving had no alert, and the disk warning had no owner.
• The missing relation goes into the CMDB — the database uses the backup storage — so the next rotation finds the archiver.
• An improvement adds an alert on failed archiving within 15 minutes and an owner for every disk warning.
• The knowledge base gets a runbook for a growing WAL. Its first line: never delete it by hand.

What made the difference

• The change record answered "what changed four days ago?" in seconds.
• The agent explained the mechanism, not just the symptom, so nobody reached for rm at 3 a.m.
• The fix covered the night and the cause: disk, archiver, CMDB, alerting and ownership.