An Experiment Proving Nothing

A team writes up its first chaos experiment:

Name:     kill a payments pod
When:     Thursday 14:00, production
Action:   delete 1 of the 12 payments-api pods
Control:  the other 11 pods, compared over the same window
Success:  no pages fire within 10 minutes
Rollback: Kubernetes reschedules the pod automatically

Which defect means this cannot produce a usable result, whichever way it goes?

There is no control group, so any change observed during the window cannot be attributed to the injected fault.

Success is defined as an absence of alerts rather than a measured output of the system, so it tests the alerting configuration instead of the service, and with no steady-state metric there is also nothing that could trip an abort mid-experiment.

Killing 1 of 12 pods is too small a variable to learn from; a first experiment should remove an entire availability zone so the result is unambiguous.

Running it in production is the defect. It belongs in staging until the team has proven the rollback path.