GPU specifications

Memory, bandwidth and rough compute for the cards people actually rent, and why bandwidth is the number that decides single-stream generation speed.

Data centre cards

A100 40GB (SXM)	40 GB HBM2e · about 1.6 TB/s · about 312 TFLOPS bf16	Still widely available and cheap to rent. The 40 GB is the constraint — it will not hold a 70B model at 4-bit with useful context.
A100 80GB (SXM)	80 GB HBM2e · about 2.0 TB/s · about 312 TFLOPS bf16	The workhorse of the previous generation. No fp8 support, which matters for serving efficiency more than for training.
H100 80GB (SXM)	80 GB HBM3 · about 3.4 TB/s · about 990 TFLOPS bf16	Roughly 1.7× the A100's bandwidth and about 3× its bf16 compute. Adds fp8, which doubles compute again where the stack supports it.
H200	141 GB HBM3e · about 4.8 TB/s · same compute as H100	Same silicon, more and faster memory. The gain is almost entirely in serving larger models and longer contexts, not in training speed.
L40S	48 GB GDDR6 · about 0.86 TB/s · about 360 TFLOPS bf16	Good compute, modest bandwidth. Fine for prefill-heavy and batch work, notably slower than its FLOPs suggest for single-stream decode.
A10G	24 GB GDDR6 · about 0.6 TB/s · about 125 TFLOPS bf16	The cheap cloud inference card. Comfortable for 7-8B models at moderate context.
Blackwell generation (B200)	about 180-192 GB HBM3e · about 8 TB/s	Figures here are approximate and configurations vary. Check the specification for the exact SKU you are renting before you plan capacity on it.

RTX 4090	24 GB GDDR6X · about 1.0 TB/s · about 165 TFLOPS bf16	More bandwidth than an A10G and more than a L40S. Excellent for local work; licence terms restrict data centre deployment.
RTX 3090	24 GB GDDR6X · about 0.94 TB/s	Still the cheapest route to 24 GB, and it supports NVLink in pairs, which the 4090 does not.
Apple Silicon, unified memory	about 0.4-0.8 TB/s depending on chip tier	Large memory capacity at modest bandwidth. Good for running big models slowly at batch 1; poor for training or for serving concurrency.

Decode is memory-bound	every generated token reads all the model weights from memory	At batch 1 the arithmetic intensity is about one FLOP per byte loaded. These cards need hundreds of FLOPs per byte to saturate their compute, so the tensor cores idle.
Single-stream ceiling	tokens per second ≈ memory bandwidth ÷ model size in bytes	A 14 GB model on a 2 TB/s card cannot exceed about 140 tokens per second, whatever its FLOPs. Real throughput lands at 60-80% of that.
Quantization speeds up decode	halving bytes per weight roughly doubles the bandwidth ceiling	This is why 4-bit models feel fast at batch 1 even where the kernels do no less arithmetic.
Batching moves you to compute-bound	weights are read once and reused across the batch	Which is why serving throughput per GPU climbs steeply with batch size while per-stream speed falls, and why offline batch work is cheap.
Prefill is compute-bound	the whole prompt is processed in parallel, so FLOPs do matter here	A card with strong compute and weak bandwidth looks good on long prompts and bad on long generations. Benchmark the mix you actually serve.

Figures last checked 2026-09-13. Several of these age; check anything you are about to rely on.

Where these numbers come from