Backpressure makes a fast producer respond to a slow consumer
Backpressure is feedback that makes a producer reduce its output when a downstream consumer cannot keep up. A bounded buffer gives the consumer time to absorb a burst. When the buffer fills, the system must slow production, reject work or discard selected data. Those choices have different consequences for correctness.
A log shipper shows why the choice is necessary. The application generates 30,000 log lines a second, while the search cluster indexes 20,000. Unsent lines accumulate at 10,000 a second. At 500 bytes per line, their payload alone grows by 5 MB a second, before object overhead and the shipper’s other memory use.
With 2 GB available exclusively for that backlog, the payload fills it in about 400 seconds. A real process can reach its limit earlier. The search cluster can be functioning normally throughout: its capacity is simply below the offered load. The shipper must make the mismatch visible upstream or apply an explicit loss policy before memory exhaustion chooses that policy accidentally.
producer 30,000 lines/s, consumer 20,000 lines/s, 500 bytes a linequeued payload grows 5 MB/s and fills 2 GB in about 400 s, before other memory costsA buffer stores the difference between arrival and service rates
The backlog changes by arrivals minus completed work. When the log rate exceeds indexing capacity by 10,000 lines a second, every extra second adds another 10,000 waiting lines. If the burst lasts 10 seconds and arrivals then fall to 10,000 a second, the consumer has 10,000 a second of spare capacity and clears that 100,000-line backlog in another 10 seconds.
Waiting time grows with the backlog. If 100,000 lines are already ahead of a newly accepted line and indexing continues at 20,000 a second, those preceding lines require about 5 seconds of service. This estimate assumes steady capacity and a first-in, first-out consumer. A queue can fit in memory while still violating a freshness target by minutes. Little's law states the same relation for any stable system: the average number of items waiting equals the arrival rate times the average time each spends inside, so at a steady 20,000 lines a second, a backlog that averages 100,000 lines means an average stay of 5 seconds.
Choose a capacity from the largest useful wait and the expected service rate, then add an appropriate burst allowance. A full buffer must trigger a defined action. Blocking preserves accepted work only when the producer can actually wait and no upstream buffer is silently growing. Rejecting a new request preserves a bounded amount of existing work, while dropping old telemetry favors freshness over completeness.
The log shipper’s requirements determine the policy. Audit records may need a durable local spool with an explicit storage limit and failure alarm. A dashboard sampling CPU temperature may discard older samples. Pausing the application whenever ordinary logging slows can make the logging system an availability dependency, so blocking is a product decision as well as an implementation detail.
Run a producer faster than its consumer
What happens to memory, waiting time and lost work when a producer offering 3 items a tick feeds a consumer that takes 2, through an 8-slot buffer? Run the same overload under each full-buffer policy, then change the two rates to find when the buffer stays empty.
After 30 ticks the unbounded buffer holds 32 items and grows by one each tick until the process runs out of memory. Each of the three bounded policies holds 8. Block made the producer hold back 24 items, preserving them upstream while its admitted rate matched the consumer's rate. Drop newest threw away 24 new items. Drop oldest threw away 24 old ones, and what the consumer took had waited 3 ticks instead of 4.
When the consumer is at least as fast as the producer, every policy behaves the same, because the buffer never fills.
Four answers to a full buffer
| Policy | The producer | What is lost | Fits |
|---|---|---|---|
| Unbounded | never waits | everything queued, when memory runs out and the process crashes | only when the total output is known to be small |
| Block | waits for downstream capacity | no intentional discard at this buffer | file copies and producers that can wait |
| Drop newest | receives a rejection or loses the new item | newly offered work | request admission with a clear 429 or 503 |
| Drop oldest | continues producing | older queued items | live positions or frames where freshness has priority |
Drop newest is what load shedding means for a web service: refuse the request you cannot finish, quickly, so the ones you accepted still finish on time. Drop oldest fits data where a newer value replaces an older one. A rider tracking a driver needs the latest position; a position from 40 seconds ago is superseded by the newer samples queued behind it, so discarding it loses nothing the rider would see.
Propagate capacity signals through each asynchronous boundary
A signal protects memory only if the producer responds to it. Suppose the shipper pauses its socket reader but its parser keeps appending decoded lines to another list. The socket buffer may remain bounded while that list grows. Inventory each queue between the original source and the final sink, including client-library buffers and tasks waiting for thread-pool execution.
Node.js writable streams provide a concrete contract. A false result from writable.write means the writer should stop submitting chunks and wait for drain. The high-water mark is a threshold for that feedback, rather than a hard guarantee that the buffer can never grow beyond it. A loop that ignores the return value defeats the mechanism even though it is using the streams API.
Demand-based protocols express capacity before sending. In Reactive Streams, a subscriber requests a number of elements and the publisher must respect the outstanding demand. TCP’s receive window limits bytes outstanding toward the receiver; it does not prove the application completed those bytes’ business work. A full Go buffered channel can block a sender, while a bounded Java executor can reject tasks or run them on the submitting thread to slow submission.
Kafka’s pull model lets a slow consumer leave unread records on the broker instead of receiving them into unlimited consumer memory. That shifts the backlog to durable storage; it does not remove it. Broker disk and retention remain finite, and an overloaded consumer can fall behind retention. Track lag and the age of the oldest required record before its recovery window closes.
Across an HTTP boundary, use a short admission wait and a clear rejection when capacity is unavailable. A 503 response with Retry-After can communicate temporary service overload; 429 identifies a client rate limit. Clients must honor the signal with bounded, randomized retries. Without that response, rejected work can immediately return and raise the arrival rate again.
What stays true
A bounded pipeline must keep admitted work within its available capacity. A sustained excess of arrivals requires lower admission, higher service capacity or an explicit discard policy.
Measure queued work together with its age and the producer’s response to a full buffer. A stable queue length caused by silent discards is a different result from successful processing.
Backpressure mistakes that appear under sustained load
Making the queue bigger to fix overload.
- Where it breaks
- In the simplified log-shipper calculation, 20 GB instead of 2 GB stores the excess payload for about 4,000 seconds instead of 400. At that point a new line waits over half an hour before it is indexed.
- The fix
- Bound the buffer and choose block or drop. If the overload is permanent, add consumers.
Alerting on queue length alone.
- Where it breaks
- A queue of 1,000 messages is a 0.1-second wait for a consumer doing 10,000 a second and a 100-second wait for one doing 10 a second. The same alert fires for both or for neither.
- The fix
- Alert on the age of the oldest message, or on consumer lag measured in time.
Blocking request threads on a full queue with no timeout.
- Where it breaks
- All 200 worker threads of a server wait on the same full queue. It stops answering health checks, the load balancer marks it dead and moves its traffic to the other servers, which fill up the same way.
- The fix
- Wait briefly, then reject with 503 and
Retry-After. Keep health checks off the busy pool.
Retrying rejected work immediately.
- Where it breaks
- Clients that get a 503 and retry at once double the arrival rate at the moment the consumer is already behind.
- The fix
- Retry with exponential backoff and jitter, under a retry budget.
Backpressure in queues, limits and retries
Related topics
- Message queues →
Every queue needs a full-buffer policy, and the dead letter queue is a drop policy for poison messages.
- Rate limiting →
A rate limiter is load shedding decided per client, before the work reaches the queue.
- Retries, backoff and jitter →
Backoff is how a client slows down when it gets a backpressure signal.
- Circuit breakers and timeouts →
A breaker sheds load toward a dependency that is failing, instead of a consumer that is slow.
Further reading
- Node.js: Backpressuring in streams What happens inside a pipe when the reader is slower than the writer. nodejs.org (opens in a new tab)
- Reactive Streams The JVM standard for demand-driven backpressure. reactive-streams.org (opens in a new tab)
- Fred Hebert: Queues don't fix overload Why a bigger queue only delays the failure, and where to put backpressure and load shedding instead. ferd.ca (opens in a new tab)
- AWS Builders' Library: Using load shedding to avoid overload Rejecting excess requests cheaply so a server keeps its goodput near capacity. builder.aws.com (opens in a new tab)
- AWS Builders' Library: Avoiding insurmountable queue backlogs Measuring backlog by age, prioritizing fresh work and recovering from a backlog that cannot drain. builder.aws.com (opens in a new tab)