5 / 6 Backpressure, DLQ & Retries

Keeping The Member Alive

A downstream dependency degraded and the handler went from 40 ms to 9 s per record. The group now rebalances every few minutes and every member reprocesses the same records after each one. Nothing is throwing.

setting value
max.poll.records 500
max.poll.interval.ms 300000
session.timeout.ms 45000
heartbeat.interval.ms 3000

More than one option below works. Select every change that stops the eviction without losing records.

A

Hand each batch to a worker pool, pause the assigned partitions so poll returns no new records, keep calling poll on schedule for liveness, then resume and commit what the pool actually finished

B

Turn on enable.auto.commit with a 5 s interval so offsets advance while the batch is still being worked through, which ends the reprocessing after each rebalance

C

Cut max.poll.records to 25, so a worst-case batch is 225 s of work and the loop is back in poll well inside the 300 s interval

D

Raise max.poll.interval.ms to 600000 and cut max.poll.records to 50, so a worst-case 450 s batch fits, at the cost of taking ten minutes to notice a consumer that is truly wedged

E

Raise session.timeout.ms from 45000 to 600000, since the member is being evicted for missing a liveness deadline and this is the deadline it is missing