Chaos engineering best practices for resilient systems

SECTION 1

Chaos engineering: testing the resilience of distributed systems

Chaos engineering is the discipline of running controlled experiments on a system in production or staging in order to build confidence in its ability to withstand degraded conditions. Rather than waiting for a failure to occur, and it will, you trigger it in a controlled way to observe the system's real behaviour, verify the fault-tolerance mechanisms and draw lessons before an incident reaches customers.

The practice has become essential as architectures have grown more complex: microservices, Kubernetes, multi-region managed services, meshes of third-party dependencies and now AI inference chains in production. In these systems, no static reasoning can guarantee the behaviour of the whole: failure modes emerge from the interactions between components, and only experimentation can uncover them before they show up on a Friday evening.

Chaos engineering is not just another test: it is a scientific method applied to reliability. You formulate a hypothesis, design an experiment, measure, and conclude. That rigour is what separates it from simply "unplugging a server to see what happens".

The discipline also differs from classic disaster recovery exercises: where a DR test validates a known procedure once or twice a year, chaos engineering continuously questions precise hypotheses about the system's behaviour, a timeout, a retry policy, an automatic failover, and produces measurable results that feed directly into the engineering backlog. The two practices complement each other, but only the second keeps pace with daily deployments and architectures that change constantly.

SECTION 2

From Chaos Monkey to GameDays: a discipline turned standard

The principles of chaos engineering were forged at Netflix in the early 2010s, with the famous Chaos Monkey that randomly shut down instances to force teams to design fault-tolerant services. Amazon, Google and the major cloud players followed, each developing their own large-scale fault-injection practices.

### Tooling that is now mature and standardised

The discipline has since become widely accessible. LitmusChaos and Chaos Mesh, projects incubated by the CNCF, bring Kubernetes-native fault injection; Gremlin and Steadybit offer managed platforms with ready-to-use attack libraries; AWS Fault Injection Service and Azure Chaos Studio embed experimentation directly into the cloud consoles. This tooling maturity shifts the challenge: the difficulty is no longer technical, it is methodological and cultural.

GameDays, planned exercises where a team simulates a major incident in real conditions, complement the automated tooling by also training the humans: escalation procedures, crisis communication, manual failovers. The best practices that follow structure this approach end to end.

This standardisation has produced well-marked maturity paths: you start with manual experiments in test environments, progress towards automated experiments with a controlled scope, then towards recurring campaigns in production, until chaos becomes part of the delivery cycle in the same way as functional tests. Few organisations reach the last stage, but all draw immediate benefit from the first ones: the most glaring weaknesses, a non-redundant dependency, a missing timeout, a silent alert, surface from the very first experiments.

SECTION 3

Define the steady state before injecting a single failure

The first best practice is to define what is normal. Without a clear reference, it is impossible to state whether an experiment degraded the service or not. You establish a baseline from the metrics that reflect the real user experience: request success rate, 99th-percentile latency, order throughput, conversion rate.

### Formulate a steady-state hypothesis

Each experiment is written as a falsifiable hypothesis: "if we take down a third of the payment service replicas, the transaction success rate stays above 99.9%". Existing SLOs naturally provide these thresholds, which is why chaos engineering builds on an SRE practice already in place, and becomes its test bench.

### Choose user-facing metrics

The classic trap is to watch infrastructure metrics (CPU, memory) that can stay healthy while the user experience collapses. The chosen metrics must measure what customers actually experience, correlated through a unified observability platform, traces, metrics and logs linked by OpenTelemetry, now the de facto standard for instrumentation.

@cite:observabilite-et-resilience-systemes-distribues

SECTION 4

Design realistic experiments and control the blast radius

Next comes disrupting that normal with care. The best experiments reproduce plausible failure scenarios, drawn from past incidents and risk analyses: increased latency on a critical dependency, loss of a pod or an availability zone, saturation of a connection pool, an expired certificate, a degraded response from an LLM API. The question is never whether a failure will occur, but when, and whether the system will survive it gracefully.

### Limit the blast radius by design

Controlling the blast radius is the precondition for confidence. You start with a minimal scope, one service, a subset of internal users, a single region, and define automatic halt conditions before the experiment: if the error rate crosses the set threshold, the injection stops and the system returns to its nominal state without human intervention. Modern tools ship these guardrails natively (halt conditions), turning a practice perceived as risky into controlled engineering.

You also prepare the teams: experiments are announced, on-call staff informed, rollback procedures rehearsed. Every triggered failure thus becomes a documented learning experience rather than a surprise.

Realism finally comes from combining faults: serious incidents rarely stem from a single isolated failure, but from the chaining of increased latency, a poorly bounded retry and a saturated queue. Composite experiments, which inject several coordinated disruptions, reveal the cascade effects that unit resilience tests never see. An up-to-date map of dependencies between services is precious here: it lets you choose the most plausible combinations rather than multiplying scenarios at random.

SECTION 5

Testing in production: progressing through controlled stages

Testing in production remains the heart, and the most debated point, of chaos engineering. No staging environment faithfully reproduces real traffic, real data and real dependencies: only production reveals the failure modes that matter. But you do not get there in one leap.

The progression happens in stages: first experiments in a test environment to validate the tooling and the hypotheses, then in staging under synthetic traffic, then in production on a canary, a restricted subset of traffic controlled by feature flags, before widening gradually. At every stage, halt conditions and steady-state metrics accompany the experiment.

This ramp-up assumes an organisation that is ready: defined SLOs, structured on-call, blameless post-mortems and a team able to carry the practice over time. Chaos engineering is as much a team practice as a technical one.

One sign of maturity never lies: when production experiments no longer trigger a debate but simply a slot on the calendar, the practice is established. Conversely, if every single campaign requires a negotiation, it means the foundations, shared SLOs, solid observability, rehearsed rollback procedures, still deserve to be consolidated before the scope is widened any further.

@cite:creer-une-equipe-sre

SECTION 6

Automate chaos continuously in CI/CD pipelines

A one-off chaos campaign photographs a system's resilience at a single point in time; yet the system changes with every deployment. The best practice is therefore to describe experiments as code, YAML manifests versioned alongside the application, and to run them continuously: resilience suites executed in CI/CD pipelines before each release, and recurring scheduled experiments on critical services.

### Towards continuous resilience verification

This automation gives rise to the notion of continuous verification: just as unit tests guarantee functional non-regression, chaos experiments guarantee resilience non-regression. A misconfigured timeout or a retry removed by accident is caught by the pipeline, not by an incident. AI-powered observability platforms strengthen the loop: automatic signal correlation during the experiment, detection of subtle anomalies and suggestion of new hypotheses drawn from real incidents.

@cite:combler-l-ecart-sre-vers-l-observabilite-autonome

SECTION 7

Building a resilience culture that lasts

The last best practice is cultural: chaos engineering only produces value if every experiment ends with shared lessons and tracked corrective actions. Each cycle feeds a knowledge base, tested hypotheses, discovered weaknesses, applied fixes, that deepens the collective understanding of the architecture and fuels continuous improvement.

Confidence is built through regularity: carefully planned experiments, the right people available during injection windows, detailed notes and metrics tracked over time to objectify progress. The telling indicators are the reduction in mean time to detect and to recover, and the share of real incidents whose scenario had already been rehearsed in an experiment.

The most advanced organisations extend the approach beyond infrastructure: chaos on data, with unexpected schemas and outlier values injected into the flows; chaos on security, with simulated compromise of credentials or certificates; organisational chaos, by playing out the absence of a key person during an incident. The principle remains rigorously identical, formulate a hypothesis, experiment under control, measure, learn, and it is this methodological constancy that gives the discipline its value.

Building resilient digital experiences finally means combining chaos engineering with the other reliability disciplines: SLOs and error budgets, observability engineering, incident management. At Adservio, we support teams in setting up these practices, from the first GameDay to fully automated resilience suites, to durably protect their applications from service disruptions.

FAQ

Frequently asked questions