”…one of the inputs to the command was entered incorrectly…” - Amazon on S3 outage, 2017
”…an improper entry was made to a call routing table during provisioning work…” - Level 3 on nationwide VOIP outage, 2016
”…an engineer [wiped the] database directory, errantly thinking they were doing so on the secondary…” - Gitlab on customer data loss, 2017
”…operational errors…”
”…standard [deployment policy] was not followed…” “…configuration switch was incorrectly enabled…” - Microsoft on Azure outage, 2014
Command-line interfaces to production environments are both ubiquitous and terrifying. They are powerful, fast, and dangerous. Most would agree that we should at least limit – if not completely eliminate – our dependence on them, but how do we achieve that lofty goal?
At Workiva we’ve started the transition. It’s tough. It’s painful. It’s wonderful. Let’s talk about it.”