Transformer memory arithmetic

What a model costs in bytes: weights, KV cache, activations and optimiser state, and how to tell in advance whether it fits on the card you have.

Weights, by dtype

Bytes per parameter	fp32 4 · bf16/fp16 2 · fp8/int8 1 · int4 0.5	bf16 is the default for training and serving. fp16 has the same size but a much narrower exponent range, which is why loss spikes and overflow are an fp16 problem more than a bf16 one.
Weight memory	parameters (billions) × bytes per parameter = GB	The whole rule. 7B in bf16 is 14 GB; 70B in bf16 is 140 GB; 70B in int4 is about 35 GB.
Usable card memory	about 90-95% of the nameplate figure	The CUDA context, the allocator and fragmentation take the rest. Plan against 72 GB on an 80 GB card, not 80.
Serving footprint	weights + KV cache + 1-2 GB runtime overhead	Leave 10-20% headroom on top, or the first long request evicts something and latency becomes erratic.

Bytes per token	2 × layers × kv_heads × head_dim × bytes_per_element	The 2 is key and value. kv_heads is the grouped-query figure, which is often 8 while attention heads number 32 or more — that difference is most of why modern models serve long contexts affordably.
8B-class model, GQA, bf16	about 128 KiB per token	32 layers, 8 kv heads, head_dim 128: 2 × 32 × 8 × 128 × 2 = 131,072 bytes. An 8k-token sequence is 1 GiB of cache.
70B-class model, GQA, bf16	about 320 KiB per token	80 layers, 8 kv heads, head_dim 128. A single 32k conversation costs roughly 10 GiB of cache on top of the weights.
Same model without GQA	4× larger, at 32 kv heads instead of 8	512 KiB per token for the 8B shape. Older multi-head models are the reason long context used to be prohibitive.
Total cache	bytes_per_token × sequence_length × concurrent_sequences	This is what actually caps your batch size in production, not the weights. Batch 32 at 8k tokens on the 8B shape is 32 GiB.
KV cache quantization	fp8 halves it; int4 quarters it	Quality loss is usually smaller than for weight quantization, but it lands on long-context retrieval first — test on your longest inputs, not your average ones.

Mixed precision with Adam	about 16 bytes per parameter	bf16 weights (2) + fp32 master copy (4) + fp32 first moment (4) + fp32 second moment (4) + gradients (2). A 1B model needs roughly 16 GB before a single activation.
Full fine-tune of 7B	about 112 GB of state, plus activations	Which is why a full tune of a 7B model does not fit on one 80 GB card, and why sharding or LoRA is the usual answer.
SGD with momentum	about 4-6 bytes per parameter less than Adam	One moment instead of two. Rarely used for transformers, because Adam's per-parameter scaling is doing real work there.
LoRA	optimiser state on adapters only, typically under 1% of parameters	The frozen base can also be 4-bit (QLoRA), so a 7B tune fits in about 6-10 GB of weights plus activations — one 24 GB card.
Activations	roughly batch × sequence × hidden × layers × a constant of a few tens	The constant depends on how many intermediate tensors the implementation keeps. Treat it as measured, not derived: run one step and read the allocator.
Gradient checkpointing	activation memory down to roughly the square root of layer count, for about 20-30% more compute	Store activations at a few boundaries and recompute the rest in the backward pass. Almost always worth it when it is the difference between fitting and not.

24 GB (4090, A10G, 3090)	7-8B bf16 inference at short context · 13B at 4-bit · QLoRA tune of 7B	Context length is what runs you out, not the weights. 14 GB of weights leaves under 10 GB of cache.
48 GB (L40S, A6000)	13B bf16 · 34B at 4-bit · comfortable QLoRA of 13B	The common single-card serving size. One card holds a mid-size model with enough cache for real concurrency.
80 GB (A100, H100)	70B at 4-bit · 34B bf16 · full fine-tune of a model up to roughly 3B	70B bf16 needs two of these for weights alone, and realistically four once you want batch and context.
8 × 80 GB node	70B bf16 served with real batch and long context · full fine-tune of 7-13B with sharding	Above this you are in multi-node territory, where interconnect bandwidth starts deciding the design.

Figures last checked 2026-09-13. Several of these age; check anything you are about to rely on.

Where these numbers come from