You Spent the Outage Fixing the One Red Light and Nobody Was Watching the Rest
The checkout went down
at half past two.
The health check for one service
was failing.
Red.
Four of you joined the call.
Four of you
looked at that service.
Its logs.
Its last deploy.
Its connection pool.
Somebody restarted it.
Still red.
Somebody rolled it back.
Still red.
Forty minutes in,
someone from another team
asked a quiet question.
Has anyone looked at the database disk?
It was full.
The red service
was the first one to notice.
It was the messenger,
not the problem.
On 29 December 1972
Eastern Air Lines Flight 401
was approaching Miami
when a light failed to confirm
that the nose gear was down.
The crew
set the autopilot to hold height
and all three of them
turned to that light.
They took the bulb out.
They tried to get it back in.
They sent the engineer below
to look at the gear.
During all this,
the autopilot was disengaged,
probably by a bump on the controls,
and the aircraft began to descend,
slowly,
in the dark,
over the Everglades.
An altitude warning chimed.
Nobody in the cockpit
reacted to it.
The aircraft hit the ground.
More than a hundred people died.
The bulb had burned out.
The gear was down.
The investigators' central finding
was that the crew
had failed to monitor the flight instruments
while they were absorbed
in the malfunction.
Under stress,
attention narrows.
It does not feel like narrowing.
It feels like focus.
The one red thing
gets sharper,
and everything else
quietly leaves the room.
That is why aviation
changed the rule.
Someone always flies the plane.
Do the same on your incidents.
One person leads
and does not touch a keyboard.
Their job
is the whole picture.
Every fifteen minutes
they ask out loud
what else could explain this.
Someone checks the shared things.
Disk.
Certificates.
DNS.
The database.
Even when the alert
points somewhere else.
And when everyone on the call
is looking at one screen,
take that as the signal
to look at another.
The fault is not always
where the light is.
– Serguey Asael Shinder