FlashAttention

In short: FlashAttention is a method that performs the attention computation in a memory-saving way, making long contexts faster and possible with less memory.

In more detail: Instead of writing the large attention matrix completely to memory, FlashAttention computes in small blocks in fast on-chip cache. The result is mathematically the same, but memory access drops strongly.

In Depth

• Effect: less VRAM at long context, faster prefill.
• Use: in llama.cpp, PyTorch, vLLM and others as an option or default.
• Requirement: newer cards and suitable software versions.

See also: Transformer architecture, Prefill and decode