Short Write Long Night

This length-prefixed framing helper has passed every test with small frames. In production it starts returning -1 for large frames even though the socket is healthy and the peer is alive. The socket is blocking and carries SO_SNDTIMEO of five seconds.

/* returns len on success, -1 on failure */
ssize_t send_frame(int fd, const uint8_t *buf, size_t len)
{
    uint32_t n = htonl((uint32_t)len);
    if (send(fd, &n, 4, 0) != 4)
        return -1;
    if (send(fd, buf, len, 0) != (ssize_t)len)
        return -1;
    return (ssize_t)len;
}

A 2 MB frame is sent over a slow link. What actually happened, and what breaks next?

Nagle coalesced the prefix with the payload into one segment, destroying the frame boundary.

send() returned a positive count smaller than len once the socket send buffer filled. Those bytes are already in the stream, so the peer reads a truncated frame and then treats payload bytes as the next length prefix.

send() failed with EPIPE because the peer closed while the transfer was in progress.

htonl truncated the length, because a 2 MB value does not fit the 4 byte prefix.