Companies embrace DevOps, working in cross-functional teams that create components that make up the microservice architectures that they live in and deliver value to its customers in. Automation, Observability, Monitoring, Collaboration and CI/CD are the enablers to be successful in this fast pace world in where we expect to deliver meaningful software deliveries multiple times per sprint.

But a lot of times we lack Performance Testing and Performance Analysis Automation and this slows us down in our delivery and is impacting our reliability negatively. In this talk I will explain everything you need to know based on my personal journey in order to become successful and enable your teams to rise above their expectations and gain speed and save precious developer time and reduce cognitive load at the same time.

I will start with explaining the principles behind it and will cover the basic DevOps principles, the importance of reaching frequency and rhythm moving away from only release based testing and touch upon the monitoring and observability needs. Then I will touch upon the need of automation and how to do this and why you want to move away spending countless of hours looking at graphs from the load test tool itself towards integrating with the standard observability and monitoring solutions available.

When you got this covered you are ready for the next step, the actual automation of the performance analysis. I will explain static and dynamic criteria, benchmarks, antipatterns, service level validation and coverage with a concrete use case.

But on which metrics do you apply it on, which data is suitable for this? I will touch upon the metrics that counted for us based on the 4 golden signals of observability which are latency, traffic, errors and saturation. We introduced Performance contracts & budgets on latency and success rates based on Googles SRE concept of SLI’s, SLO’s and Error Budgets.

By now the heavy lifting is done, but you are still not there yet, next on your list are the topics of evidence/reporting and don’t forget the feedback loop towards production which is one of the big bonusses using the standard observability and monitoring solutions. It enables you to compare the metrics of the load tests with production data and give you valuable insights on the coverage of components and endpoints and possible differences on load, latency and success rate among other metrics. It enables you to score a load test coverage score on which you can steer that tells you when you need to improve the coverage of your load tests or the quality of your test setup in general.

The last part is about organizational aspects and culture, how do you adopt this within the organization to be successful and what are the pitfalls, giving some tips about first steps to close it off.

By now I hope your spark is ignited and you are setup for success to take your first steps towards automating performance testing and analysis. It is a fun journey and there might be specific challenges you need to tackle, the rewards are waiting for you.

Stephan’s passion is helping making ING’s IT systems meeting our customers’ expectations. This is our license to operate and our promise to our customers. Stephan started in 2005 and