Question 871

Counting the Wasted Slots

vLLM allocates the KV cache in fixed blocks of 16 tokens. A batch holds four sequences of length 39, 97, 130, and 200 tokens, on a model whose cache costs 800 KB per token. The system it replaces reserved one contiguous buffer per sequence, sized to the 2,048-token maximum.

How much does each scheme waste right now?

A

64 slots with paging, one whole unused block per sequence, against 7,726 slots reserved.

B

7,726 slots with paging and 46 with the contiguous buffers, because per-block rounding is what creates the waste.

C

46 token slots (about 37 MB) with paging, against 7,726 slots (about 6.2 GB) reserved and unused with max-length buffers. Paging's waste is bounded by the last partial block of each sequence.

D

Nothing with paging: blocks are allocated on demand as the sequence grows, so no space is left unused.

E

46 slots with paging plus roughly as much again in external fragmentation, since blocks from finished sequences leave gaps the allocator cannot reuse.