Blue screens everywhere: a faulty security update and a major incident

07:05: the first tickets

People arrive at work, open their laptops and get a blue screen, then a restart loop. In ten minutes six teams call the service desk. The Triage Agent sees what one person on the phone can't: six tickets, six teams, the same crash, all on Windows laptops running the same endpoint security agent — which pushed a content update at 06:40. It groups the tickets under one incident and recommends a major incident.

Major incident, and people first

• At 07:18 Deniz declares a severity-one major incident and takes command.
• The first message goes to the whole company straight away: if your laptop works, don't restart it. Macs are fine. Next update at eight.
• At 07:35 the vendor confirms a faulty content update, withdrawn at 07:22, with a fix that needs safe mode.

Exactly who is affected

The Ops Assistant works out the scope from Assets and device management: 31 laptops run the agent; 19 were online during the 42 minutes the bad update was live; the other 12 got the corrected content and are fine. Safe mode needs each laptop's BitLocker recovery key, so the assistant prepares the list — and the service desk reads each key out at the desk or by phone. Nobody emails a key.

Desk by desk

• The work is split three ways: two office floors, and remote staff guided by phone — about ten minutes per laptop.
• Every 30 minutes a status update goes out. At 09:30: 14 of 19 fixed.
• At 10:40 all 19 laptops work again. The major incident is resolved, and the six tickets close with it.

So it can't happen the same way again

• The problem record names the real issue: the vendor's updates reach every laptop at the same moment, so a bad one hits everyone before anyone can stop it.
• Five IT laptops now get updates two hours before everyone else, with an alert if two of them go quiet.
• The safe-mode fix goes into the knowledge base, key handling included, and the management summary is ready before lunch.