Step 1: weights

Llama 3.1 70B has 70.6B parameters.

• BF16: 70.6B x 2 bytes = 141.2 GB (131.5 GiB)
• FP8/INT8: 70.6B x 1 = 70.6 GB (65.8 GiB)
• INT4: 70.6B x 0.5 = 35.3 GB, plus a few percent for quantization scales

Step 2: KV cache per token

From the model's config: 80 layers, 8 KV heads (grouped-query attention), head_dim 128.

KV per token = 2 x 80 x 8 x 128 x 2 bytes = 327,680 bytes = 320 KiB (BF16), or 160 KiB with an FP8 KV cache.

Per sequence:

• 4,096 tokens: 1.25 GiB
• 8,192 tokens: 2.5 GiB
• 32,768 tokens: 10 GiB
• 128K tokens (the model's maximum, 131,072): 40 GiB

Step 3: overhead

Engines reserve memory for activations, CUDA graphs, and the CUDA context. Budget roughly 1 to 3 GiB per GPU, and remember engines like vLLM only use a fraction you set (--gpu-memory-utilization, default 0.9).

Step 4: put it together

Usable memory per 80 GB GPU at 0.9 utilization ~= 72 GB = 67 GiB, minus ~2 GiB overhead = ~65 GiB.

(Numbers from the formulas above; "8K-token sequences" = KV left / 2.5 GiB.) The FP8 row shows why "70B in FP8 fits on one H100" is only just true: you need to raise utilization close to 1.0 and keep context short. A 141 GB H200 holds FP8 70B comfortably (~50 GiB left for KV at 0.9 utilization).

Step 5: choose by workload

• Few users, short prompts, budget-limited: INT4 on one 80 GB GPU.
• Production chat at moderate scale: BF16 or FP8 with tensor parallelism across 4 GPUs, or FP8 on 2 GPUs.
• Long context (32K+) for many users: KV dominates; use FP8 KV cache, more GPUs, and prefix caching.

Do it for any model

Read config.json and use:

• weights = params x bytes per param
• KV/token = 2 x num_hidden_layers x num_key_value_heads x (hidden_size / num_attention_heads) x bytes
• concurrent tokens = (usable memory - weights) / KV per token

Our zero-dependency calculator does this for you: kv_calc.py.

If you want to go deeper, the free Foundations course at Inference Engineering Academy covers these topics with browser labs. Disclosure: written by the academy team.

Two phases

Prefill runs the whole prompt through the model in one forward pass. Every prompt token is processed in parallel, so each layer does matrix × matrix multiplications. Prefill fills the KV cache and produces the first output token, so it drives time to first token (TTFT).

Decode then generates one token per step. Each step pushes a single new token (per sequence) through every layer and attends over the cached K/V. Decode drives time per output token (TPOT), also called inter-token latency.

Arithmetic intensity

A GPU is limited either by how fast it can compute (FLOP/s) or by how fast it can read memory (bytes/s). The ratio that decides which one applies is arithmetic intensity: FLOPs done per byte moved.

An H100 SXM offers roughly 989 TFLOP/s of dense BF16 and about 3.35 TB/s of HBM bandwidth. The break-even point (the "ridge" of the roofline) is about 989e12 / 3.35e12 ≈ 295 FLOPs per byte. Below that, the work is memory-bound.

For a matrix multiply against a weight matrix, each BF16 weight (2 bytes) gets used for 2 FLOPs (multiply + add) per token in the batch:

• Decode, batch 1: ~2 FLOPs per 2 bytes ≈ 1 FLOP/byte, far below 295. The GPU spends most of its time waiting for weights to arrive from HBM.
• Prefill, 2,000-token prompt: each weight is reused for 2,000 tokens, so ≈ 2,000 FLOPs/byte, well above the ridge. That's compute-bound.

Back-of-envelope decode speed

Llama 3.1 8B in BF16 is about 16 GB of weights. Every decode step has to read all of them at least once:

16 GB / 3.35 TB/s ≈ 4.8 ms per step  →  ≈ 200 tokens/s upper bound (batch 1)

Real numbers come in lower because of KV-cache reads, kernel overheads and imperfect bandwidth use. Notice that FLOPs never showed up in that estimate. For single-stream decode, memory bandwidth sets the speed.

Why batching is the main lever

If 32 sequences decode together, the weights are read once per step and used 32 times. Intensity rises about 32×, so throughput rises almost linearly while per-step latency barely moves. That's why serving engines use continuous batching: they admit new requests into the running batch at every step instead of waiting for a whole batch to finish.

There's a catch. Weights are shared across the batch, but KV cache is not: each sequence reads its own cache on every step. As batch size and context grow, cache reads take over and decode stays memory-bound. This is why KV-cache size (128 KiB/token for Llama 3.1 8B in BF16) matters for speed as well as capacity.

What follows from this

• Quantizing weights speeds up decode because there are fewer bytes to read. INT4/FP8 weights help decode more than they help prefill.
• Speculative decoding uses spare compute: a small draft model proposes several tokens and the big model checks them in one pass, which is more work per byte read.
• Chunked prefill splits long prompts into pieces so a big prefill doesn't stall everyone else's decode steps.
• Prefill/decode disaggregation (as in NVIDIA Dynamo, and supported by vLLM and SGLang) runs the two phases on separate GPU pools, because they want different hardware and different batch sizes.
• Measure TTFT and TPOT separately. A change that helps one often hurts the other.

Summary

Prefill is a big parallel matrix multiply and is compute-bound. Decode reads every weight and the whole KV cache to produce one token, so it is memory-bandwidth-bound. Almost every serving optimization (batching, quantization, paging, speculation, disaggregation) is a way of getting more useful tokens out of each byte read from memory.

If you want to go deeper, the free Foundations course at Inference Engineering Academy covers prefill, decode and the KV cache with runnable labs. Disclosure: written by the academy team.

What the KV cache stores

In every attention layer, each token produces a key vector and a value vector. During decoding, a new token's query has to attend over the keys and values of all earlier tokens. Because of the causal mask those earlier K/V vectors never change, so engines compute them once and keep them in GPU memory. That stored state is the KV cache. Without it, every decode step would recompute the whole prefix.

The formula

bytes per token = 2 × num_layers × num_kv_heads × head_dim × bytes_per_element

• 2: one key plus one value
• num_layers: num_hidden_layers
• num_kv_heads: num_key_value_heads. This is the number of KV heads, not query heads.
• head_dim: head_dim, or hidden_size / num_attention_heads if the field is missing
• bytes_per_element: 2 for BF16/FP16, 1 for FP8

Llama 3.1 8B, worked out

From its config: 32 layers, 32 query heads, 8 KV heads (grouped-query attention), hidden size 4096, so head_dim = 4096 / 32 = 128. At BF16:

2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KiB per token

Now scale it:

The BF16 weights of the 8B model take about 16 GB (~15 GiB). A single full-context request needs more memory for cache than the model needs for weights. A model that "fits" on a 24 GB card can still go out of memory as soon as real traffic arrives.

The most common mistake: using query heads

If you plug in 32 attention heads instead of 8 KV heads, you get 512 KiB per token, which is 4× too high. Grouped-query attention lets 4 query heads share each KV head, and that sharing is exactly why Llama 3 can offer long contexts. For Llama 3.1 70B (80 layers, 8 KV heads, head_dim 128) the same formula gives 320 KiB per token, about 40 GiB at full 128K context.

What this means when you serve

1. Concurrency budget. Free memory after weights ÷ (bytes per token × average sequence length) ≈ how many requests you can hold at once. On an 80 GB GPU running the 8B model with roughly 60 GB left for cache, that's about 480K tokens in total: ~60 requests at 8K, but only 3 at full context.
2. FP8 KV cache halves it. At 1 byte per element you get 64 KiB per token. Engines like vLLM and SGLang support this; check quality on your own workload.
3. Paging matters. Engines that preallocate the maximum length waste most of this memory. PagedAttention (vLLM) and radix-tree caches (SGLang) allocate in blocks, so memory follows actual length.
4. Decode speed falls with context. Every decode step reads the whole cache for that sequence, so long contexts cost bandwidth as well as capacity.

A quick calculator

def kv_bytes_per_token(layers, kv_heads, head_dim, dtype_bytes=2):
    return 2 * layers * kv_heads * head_dim * dtype_bytes

per_tok = kv_bytes_per_token(32, 8, 128)       # Llama 3.1 8B
print(per_tok / 1024, "KiB/token")              # 128.0
print(per_tok * 131072 / 2**30, "GiB @ 128K")   # 16.0

Run this for any model before you choose a context length, a GPU, or --max-num-seqs.

If you want to go deeper, the free Inference Engineering Foundations course at inference-engineering.xyz works through these numbers for Llama 3.1 8B with runnable NumPy code.