Chaos Engineering: Experiments That Prove Detection

Chaos engineering is the practice of breaking part of a running system on purpose, while you are watching, to learn how the rest of it behaves. The breakage is not the point. The point is the hypothesis you wrote down beforehand, and the measurement that either survives the fault or does not.

What is chaos engineering?

The discipline is experimentation on a live system to build confidence that it withstands the conditions production actually throws at it. The Principles of Chaos manifesto set out the formulation most teams still use. It started at Netflix, whose Chaos Monkey terminated production instances at random during office hours. The theory was simple. A failure found at 2pm costs less than the same failure at 3am.

An experiment has four parts, and skipping any of them turns it into vandalism.

1. Define a steady state as a number you already collect.
2. Predict that the number holds while a specific fault is present.
3. Inject that fault under real traffic, with a stop condition ready.
4. Try hard to disprove the prediction, then write down what happened.

How does an experiment actually run?

You pick one fault, bound who it can reach, and set a trigger that ends the run early. Blast radius is the whole craft here. Killing one replica of a three-replica service is an experiment, and killing the database is an outage with a nicer name.

Fault to inject	What it is meant to prove	Usual blast radius
Terminate one instance	The scheduler replaces it and the load balancer drains it	One pod or node
Add 200 ms of latency to a dependency	Timeouts and retries are set sanely	One service pair
Return errors from a third-party API	The fallback path exists and works	One integration
Take one availability zone offline	Capacity survives without it	One zone, off peak

Tooling is mature and mostly open. Chaos Mesh and LitmusChaos run inside Kubernetes, and Gremlin sells a hosted agent. AWS Fault Injection Service drives faults against AWS resources, with stop conditions wired to CloudWatch alarms as of September 2026.

The steady state has to be a number

Before you touch anything, record what normal looks like. A twenty sample baseline against the endpoint you care about takes twenty seconds and settles arguments later.

for i in $(seq 1 20); do
  curl -s -o /dev/null -w "%{http_code} %{time_total}\n" https://example.com/checkout
  sleep 1
done

Write the hypothesis in terms of that output. Something like: status stays 200 and p95 stays under 800 ms while one of three app instances is gone. Now the experiment has a pass condition that nobody can argue with afterwards.

Who proves the experiment was detected?

This is the part page one of the search results skips. Your dashboards live in the same account, cluster and network you are deliberately damaging. If the fault takes out the path that carries your telemetry, the graphs go quiet and quiet reads as healthy.

An independent outside-in check is the referee. Point a 30 second check at the same URL your customers use, run it from two regions, and compare its verdict with your internal one. Three outcomes are all useful. The probe stayed green and so did the dashboards, so the system absorbed the fault. The probe went red and the dashboards did not, so detection is broken. The probe stayed green while the dashboards screamed, so the alert is tuned too tight.

We prefer that third outcome to surface on a Tuesday afternoon rather than during a real incident. That is the whole argument for the method. The API reference covers creating the check, and the egress addresses to allowlist during a network fault sit with the probe regions.

There is a real limit to what the referee can say. An external probe reports that users were affected, not which pod died. Pair it with whatever instrumentation runs inside the process.

Is chaos engineering still worth doing?

The honest answer depends on what you have built. A sceptical r/devops thread argues that most teams break under faults they could have predicted on a whiteboard. On that reading, randomised failure injection is a luxury for systems with redundancy worth testing.

That reading is fair. If you run one application server and one database, the useful experiment is failover testing, which asks whether the standby takes over at all. Randomised fault injection earns its place once there is enough redundancy that nobody can say with confidence what a single loss does.

Common mistakes

Running in staging and calling it done. Staging has no real traffic, no real data volume and usually no real network topology. A fault that passes there proves the fault injector works.

No stop condition. Every run needs a threshold that ends it automatically and a human who can end it faster. Without one, an experiment becomes an incident with an audience.

Never checking the detection path. The most common finding from a good chaos engineering run is not that the system fell over. It is that nobody would have known for eleven minutes.

Start small and start with detection. Kill one instance during a quiet hour, watch an independent check confirm what your own alerting claims, and write the result down even when nothing breaks. That boring result is the one worth having. Chaos engineering only pays off when the measurement is trustworthy, and a second opinion from outside your own network costs nothing to set up on the free plan.