Why does continuous batching raise throughput so much?

An answer that works

Decode is memory-bound, so one step reads the weights once regardless of how many sequences are in the batch, and extra sequences use arithmetic that would otherwise be idle. Continuous batching keeps that batch full by admitting and retiring sequences at every step, rather than letting the whole batch wait for its longest member on mixed-length traffic.