Speculative decoding

In short: A small, fast model guesses several tokens ahead, and the large model checks them all at once. If the guess is right, several tokens arise in one step.

In more detail: Because checking several tokens takes almost as long as generating a single one (the weights are read once anyway), speed rises by a factor of 1.5 to 3 when the small model is often right. The result is identical to that of the large model.

In Depth

• Good for: predictable text (code, standard answers).
• Less good: creative and random.
• Variant: Medusa and EAGLE use extra heads instead of a small model.
• Memory: The small model needs extra space.

See also: Tokens per second, Prefill and decode