Mixture of Experts (MoE)

In short: In a mixture-of-experts model, every layer consists of many “experts” of which only a few are active per token. This makes it much faster than a “dense” model of the same size.

In more detail: A model may, for example, have 100 billion parameters in total but use only 12 billion per token. It needs memory for all 100 billion but reads only the active part per token. Speed corresponds to a 12-billion model.

Our context: The AI speed calculator distinguishes “total parameters” and “active parameters” and accounts for more experts being read when several requests are batched.

In Depth

• Router: A small network chooses the matching experts per token.
• Advantage: more knowledge at the same compute per token.
• Disadvantage: The full memory need remains. If the model does not fit in VRAM, offloading helps well (offloading) because only parts are read from RAM per token.
• Examples: Mixtral, DeepSeek-V3, Qwen3-MoE, GPT-OSS (depending on version).

See also: Tokens per second, Parameters (AI model)