Forty Milliseconds Exactly

An RPC client on an established, idle connection issues two write() calls per request: 4 bytes of length prefix, then a 96 byte body. The server parses the length, reads the body, and only then replies.

Measured client-side latency over an hour on a 0.4 ms LAN:

percentile	latency
p50	0.4 ms
p90	0.4 ms
p99	40.1 ms
p99.9	40.2 ms

There are zero retransmissions in the capture and zero packet loss on the switch counters. What produces the 40 ms cliff?

The server's accept backlog filled, so the SYN was dropped and the connection was retried.

Nagle holds the 96 byte write because the 4 byte write is still unacknowledged, and the server has no reply to piggyback on, so its delayed ACK timer runs to completion.

The retransmission timer fired and the second segment was resent.

Congestion control halved cwnd after a loss the switch counters did not record, so the second write waited for a window to open.