Title: One engineer, four environments, no termination policy.

Description:

Terminating all environments just days before going live and getting back from scratch. We were days away from going live with our AWS Mesos stack until one unlucky engineer ran a faulty playbook.

This talk will focus on how we build and operate the new Wehkamp stack. We’ll start at the AWS level where we have several hundred instances, ELB’s and supporting services. On top of those instances we run services like Mesos, Consul, Kafka, ElasticSearch and Cassandra. Our real workload lives on Mesos, the microservices architecture we call Blaze. Next we’ll go over what happened in the days before we flipped the switch to go live, and the steps involved creating everything from scratch. How we utilize ansible and we’ll briefly touch Jenkins as our CI/CD tool. The final words will describe our plans in this area for the near future, and the improvements on what we’ve seen so far.