Shedding Looks Like Success

This function defines the availability SLI for an API. During the evening peak the service starts shedding load and returns 429 Too Many Requests to about 8 percent of traffic, but the SLO dashboard stays at 99.98 percent while support tickets pile up.

def is_good(req):
    return req.status < 500

sli = sum(is_good(r) for r in requests) / len(requests)

What is the right correction?

Switch from a request-ratio SLI to time-based availability, measuring the minutes in which the service was up

Count 429 as a failure while leaving 4xx caused by malformed client requests as successes, since a 429 is your service declining to serve a well-formed request

Count every 4xx as a failure, because users experience all of them as the service not working

Leave the SLI alone and put the 429 rate on its own dashboard with a separate alert