Messaging and streams 0/11

Lesson 2 of 11

Backpressure

Decide what happens when a producer outruns its consumer: buffer, block, drop or shed load.

1,603 words · 8 min read Useful background: Message queues

Backpressure makes a fast producer respond to a slow consumer

Backpressure is feedback that makes a producer reduce its output when a downstream consumer cannot keep up. A bounded buffer gives the consumer time to absorb a burst. When the buffer fills, the system must slow production, reject work or discard selected data. Those choices have different consequences for correctness.

A log shipper shows why the choice is necessary. The application generates 30,000 log lines a second, while the search cluster indexes 20,000. Unsent lines accumulate at 10,000 a second. At 500 bytes per line, their payload alone grows by 5 MB a second, before object overhead and the shipper’s other memory use.

With 2 GB available exclusively for that backlog, the payload fills it in about 400 seconds. A real process can reach its limit earlier. The search cluster can be functioning normally throughout: its capacity is simply below the offered load. The shipper must make the mismatch visible upstream or apply an explicit loss policy before memory exhaustion chooses that policy accidentally.

Inputproducer 30,000 lines/s, consumer 20,000 lines/s, 500 bytes a line
Outputqueued payload grows 5 MB/s and fills 2 GB in about 400 s, before other memory costs

A buffer stores the difference between arrival and service rates

The backlog changes by arrivals minus completed work. When the log rate exceeds indexing capacity by 10,000 lines a second, every extra second adds another 10,000 waiting lines. If the burst lasts 10 seconds and arrivals then fall to 10,000 a second, the consumer has 10,000 a second of spare capacity and clears that 100,000-line backlog in another 10 seconds.

Waiting time grows with the backlog. If 100,000 lines are already ahead of a newly accepted line and indexing continues at 20,000 a second, those preceding lines require about 5 seconds of service. This estimate assumes steady capacity and a first-in, first-out consumer. A queue can fit in memory while still violating a freshness target by minutes. Little's law states the same relation for any stable system: the average number of items waiting equals the arrival rate times the average time each spends inside, so at a steady 20,000 lines a second, a backlog that averages 100,000 lines means an average stay of 5 seconds.

Choose a capacity from the largest useful wait and the expected service rate, then add an appropriate burst allowance. A full buffer must trigger a defined action. Blocking preserves accepted work only when the producer can actually wait and no upstream buffer is silently growing. Rejecting a new request preserves a bounded amount of existing work, while dropping old telemetry favors freshness over completeness.

The log shipper’s requirements determine the policy. Audit records may need a durable local spool with an explicit storage limit and failure alarm. A dashboard sampling CPU temperature may discard older samples. Pausing the application whenever ordinary logging slows can make the logging system an availability dependency, so blocking is a product decision as well as an implementation detail.

Run a producer faster than its consumer

What happens to memory, waiting time and lost work when a producer offering 3 items a tick feeds a consumer that takes 2, through an 8-slot buffer? Run the same overload under each full-buffer policy, then change the two rates to find when the buffer stays empty.

Producer3 per tick

Buffer: 0 of 8 slots used, newest on the left

Consumer2 per tickDone: 0
Tick
0
Oldest waited
0
Produced
0
Dropped
0
Held back
0

After 30 ticks the unbounded buffer holds 32 items and grows by one each tick until the process runs out of memory. Each of the three bounded policies holds 8. Block made the producer hold back 24 items, preserving them upstream while its admitted rate matched the consumer's rate. Drop newest threw away 24 new items. Drop oldest threw away 24 old ones, and what the consumer took had waited 3 ticks instead of 4.

When the consumer is at least as fast as the producer, every policy behaves the same, because the buffer never fills.

Four answers to a full buffer

What each policy does when the buffer is full
PolicyThe producerWhat is lostFits
Unboundednever waitseverything queued, when memory runs out and the process crashesonly when the total output is known to be small
Blockwaits for downstream capacityno intentional discard at this bufferfile copies and producers that can wait
Drop newestreceives a rejection or loses the new itemnewly offered workrequest admission with a clear 429 or 503
Drop oldestcontinues producingolder queued itemslive positions or frames where freshness has priority

Drop newest is what load shedding means for a web service: refuse the request you cannot finish, quickly, so the ones you accepted still finish on time. Drop oldest fits data where a newer value replaces an older one. A rider tracking a driver needs the latest position; a position from 40 seconds ago is superseded by the newer samples queued behind it, so discarding it loses nothing the rider would see.

Propagate capacity signals through each asynchronous boundary

A signal protects memory only if the producer responds to it. Suppose the shipper pauses its socket reader but its parser keeps appending decoded lines to another list. The socket buffer may remain bounded while that list grows. Inventory each queue between the original source and the final sink, including client-library buffers and tasks waiting for thread-pool execution.

Node.js writable streams provide a concrete contract. A false result from writable.write means the writer should stop submitting chunks and wait for drain. The high-water mark is a threshold for that feedback, rather than a hard guarantee that the buffer can never grow beyond it. A loop that ignores the return value defeats the mechanism even though it is using the streams API.

Demand-based protocols express capacity before sending. In Reactive Streams, a subscriber requests a number of elements and the publisher must respect the outstanding demand. TCP’s receive window limits bytes outstanding toward the receiver; it does not prove the application completed those bytes’ business work. A full Go buffered channel can block a sender, while a bounded Java executor can reject tasks or run them on the submitting thread to slow submission.

Kafka’s pull model lets a slow consumer leave unread records on the broker instead of receiving them into unlimited consumer memory. That shifts the backlog to durable storage; it does not remove it. Broker disk and retention remain finite, and an overloaded consumer can fall behind retention. Track lag and the age of the oldest required record before its recovery window closes.

Across an HTTP boundary, use a short admission wait and a clear rejection when capacity is unavailable. A 503 response with Retry-After can communicate temporary service overload; 429 identifies a client rate limit. Clients must honor the signal with bounded, randomized retries. Without that response, rejected work can immediately return and raise the arrival rate again.

What stays true

A bounded pipeline must keep admitted work within its available capacity. A sustained excess of arrivals requires lower admission, higher service capacity or an explicit discard policy.

Measure queued work together with its age and the producer’s response to a full buffer. A stable queue length caused by silent discards is a different result from successful processing.

Backpressure mistakes that appear under sustained load

  1. Making the queue bigger to fix overload.

    Where it breaks
    In the simplified log-shipper calculation, 20 GB instead of 2 GB stores the excess payload for about 4,000 seconds instead of 400. At that point a new line waits over half an hour before it is indexed.
    The fix
    Bound the buffer and choose block or drop. If the overload is permanent, add consumers.
  2. Alerting on queue length alone.

    Where it breaks
    A queue of 1,000 messages is a 0.1-second wait for a consumer doing 10,000 a second and a 100-second wait for one doing 10 a second. The same alert fires for both or for neither.
    The fix
    Alert on the age of the oldest message, or on consumer lag measured in time.
  3. Blocking request threads on a full queue with no timeout.

    Where it breaks
    All 200 worker threads of a server wait on the same full queue. It stops answering health checks, the load balancer marks it dead and moves its traffic to the other servers, which fill up the same way.
    The fix
    Wait briefly, then reject with 503 and Retry-After. Keep health checks off the busy pool.
  4. Retrying rejected work immediately.

    Where it breaks
    Clients that get a 503 and retry at once double the arrival rate at the moment the consumer is already behind.
    The fix
    Retry with exponential backoff and jitter, under a retry budget.

Backpressure in queues, limits and retries

Related topics

Further reading