Transformers

KV Caching in LLM Inference

Large language models generate text one token at a time. Without an optimization, every new token would force the model to repeat attention calculations for tokens it has already processed. KV caching avoids much of that repeated work. During inference, the model stores the key and value representations produced by attention layers for previous tokens. When generating the next token, it can reuse those stored representations instead of recomputing them from the beginning.