Asked about What modern models actually do differently
A model card says 120 billion total parameters and 5 billion active. What does that tell you about serving it?
An answer that works
It is a mixture of experts: a router picks a couple of experts per token, so compute per token is that of a 5 billion parameter model while all 120 billion must sit in memory. It will be cheap per token and expensive to host. Sizing the GPU from the active figure is the standard mistake.
Then they push further
What is harder about training it?
Keeping the router balanced, which needs an auxiliary loss term or the router collapses onto a few experts, and the all-to-all communication when experts live on different devices.
Reading an answer is not being able to give one. The section that teaches this is What modern models actually do differently, 6 hours of Transformers & attention.
Learn this properlyAlso asked about Transformers & attention