Forty Milliseconds Of Silence

An RPC client writes each request as two write calls, a 4-byte length prefix and then a 96-byte body, on a connection with a 0.5 ms round trip. It then blocks on the reply. Measured request latency is a steady 40 ms. Nagle is on at the client, delayed ACKs at the server.

What is stalling?

A 4-byte segment nearly fills the server's receive window, so flow control blocks the second write until the application drains the buffer.

Nagle holds small writes for a fixed timer before sending, and 40 ms is that timer.

The 4-byte segment goes out immediately; Nagle then holds the 96-byte body until the first segment is acknowledged, while the server's delayed-ACK timer holds that acknowledgement because it has no data to send back and only one segment to acknowledge. Each side waits for the other until the ACK timer expires.

The congestion window is one segment on a fresh connection, so the body cannot be sent until the first segment is acknowledged.