Twelve Pods Restart Together

Twelve pods serve this Go API. The handler below is wired to the Kubernetes liveness probe: httpGet on /healthz, periodSeconds: 5, failureThreshold: 2. Most endpoints on this service never query the database.

func healthz(w http.ResponseWriter, r *http.Request) {
    ctx, cancel := context.WithTimeout(r.Context(), 2*time.Second)
    defer cancel()

    if err := db.PingContext(ctx); err != nil {          // the shared primary
        http.Error(w, "db down", http.StatusServiceUnavailable)
        return
    }
    if err := recommender.Ping(ctx); err != nil {        // optional page module
        http.Error(w, "recommender down", http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
}

The shared primary stalls for 20 seconds. What happens, and what is the smallest correct fix?

Only the recommender line is a defect. Pinging the primary is exactly what a liveness probe should do, since a process that cannot reach its database cannot serve anything.

Every pod fails two probes inside the stall, so the kubelet restarts all twelve at once and endpoints that never touch the database go down too, with a cold-start reconnect storm on top. A liveness probe should report only whether this process is alive; shared-dependency health belongs in a separate signal that does not restart or drain the whole fleet at once.

About 40 percent of the pods restart, because only the ones whose probe window happens to overlap the stall accumulate two failures in a row.

Nothing restarts: the 2 second context deadline turns the stall into a returned HTTP response, and the kubelet treats any response it receives as a live process.