Efficient Memory Management for Large Language Model Serving with PagedAttention(opens in a new tab)
PagedAttention and vLLM: store each request's KV cache in fixed-size blocks found through a block table, like pages of virtual memory, which nearly removes fragmentation and lets requests share blocks.