How much GPU memory do you need to serve a 70B model?
Weights, KV cache and overhead for Llama 3.1 70B, step by step
Latest articles
Prefill vs decode: why LLM inference is memory-bandwidth bound
Why a petaFLOP GPU streams ~100 tokens/s, with the arithmetic
KV cache math: why Llama 3.1 8B needs ~128 KB per token
If you serve LLMs, the KV cache, not the weights, usually decides how many users fit on one GPU. The good news is that its size is easy to calculate from config.json, and the calculation explains a lo