6 / 6 Message Queues & Brokers

The Flat Age Alarm

A standard SQS queue has a redrive policy with maxReceiveCount set to 10 and a dead-letter queue attached. A batch of malformed jobs is failing on every attempt. An operator is paging on ApproximateAgeOfOldestMessage and the metric stays under two minutes all afternoon, so nobody notices for six hours.

receive # of the poison message what the consumer does what the queue does next
1 throws, does not delete message returns after the visibility timeout
2 throws, does not delete message returns after the visibility timeout
3 throws, does not delete ?
10 throws, does not delete moved to the dead-letter queue

Why did the age alarm stay quiet while these messages were stuck?

A

A message counts as new from the age metric's point of view each time it becomes visible again, so its age restarts at zero on every redelivery

B

For a standard queue with maxReceiveCount above 3, SQS moves a message that has been received 3 or more times to the back of the queue, and the age metric then reports the age of the oldest message that has not crossed that threshold

C

The age metric only counts messages that are visible, and the poison messages spent most of the afternoon inside their visibility timeout, so the metric never sampled them

D

ApproximateAgeOfOldestMessage is reset every time any consumer successfully deletes any message, and healthy traffic was flowing through the same queue the whole time