Question 1491

An Experiment Proving Nothing

A team writes up its first chaos experiment:

Name:     kill a payments pod
When:     Thursday 14:00, production
Action:   delete 1 of the 12 payments-api pods
Control:  the other 11 pods, compared over the same window
Success:  no pages fire within 10 minutes
Rollback: Kubernetes reschedules the pod automatically

Which defect means this cannot produce a usable result, whichever way it goes?

A

There is no control group, so any change observed during the window cannot be attributed to the injected fault.

B

Success is defined as an absence of alerts rather than a measured output of the system, so it tests the alerting configuration instead of the service, and with no steady-state metric there is also nothing that could trip an abort mid-experiment.

C

Killing 1 of 12 pods is too small a variable to learn from; a first experiment should remove an entire availability zone so the result is unambiguous.

D

Running it in production is the defect. It belongs in staging until the team has proven the rollback path.