So the dumpster is on fire. Again. The site’s down. Your boss’s face is an ever-deepening purple. And you begin debating whether you should join the #incident channel or call an ambulance to deal with his impending stroke.
Yes, we know this is a developer’s fault. There’s plenty of time for blame later. Postmortems have a macabre name because they were once intended to be Viking-like funerals for someone’s job. But we’re civilized now. Sort of. So we call them post-incident reviews.
Fires are never going to stop. We’re human. We miss bugs. Or we fat finger a command — deleting dozens of servers and bringing down S3 in US-EAST-1 for hours — effectively halting the internet. These things happen.
But we can fundamentally change the way we approach fires. And that requires adopting the techniques of industries much older than ours.
Firefighting (the kind with flames) was created as a lucrative scheme by Marcus Licinius Crassus in Ancient Rome. Only we wouldn’t have considered his brigade heroes. They would arrive to the fire and do nothing while one “firefighter” negotiated the price of their services. If an agreement could not be reached, they would let the property burn to the ground and then offer to buy it for a fraction of the original value.
This is probably not a great strategy to use with customers. Though you could pitch it to a VC — just be sure to use the word “blockchain” in your proposal.
Firefighters have clear procedures and a strong hierarchy. The first truck at a scene immediately begins assessing the situation. They evaluate building construction, visible smoke, fire and flow paths. The Incident Commander gives orders to his or her personnel regarding fire attack as well as calling for additional resources, communicating with those not yet at at the scene and preparing an action plan.
Which sounds a lot like the tight, orderly process of a deploy rollback or hotfix. Just kidding.
This talk will focus on how firefighters approach a fire — before, during and after an incident — and how we can apply those strategies to our own (admittedly much less dangerous) fires.