Flaky tests are a waste of time and money. It becomes a nightmare when the component under test is a very complex non deterministic decision system. We found that mocks, stubs and docker sidecars have their limits and we moved more tests from the CI to pre-production testing (canary on steroids)