Memory hierarchy and cache lines

In short: Computers have several memory levels: registers, L1–L3 cache, main memory, SSD, hard disk. The closer to the CPU, the faster but smaller. Data moves in cache lines (usually 64 bytes).

In more detail: Accessing main memory takes roughly a hundred times longer than L1 cache. Programs that read data sequentially and close together (spatial and temporal locality) run much faster.

In Depth

• Orders of magnitude: L1 ≈ 1 ns, L2 ≈ 4 ns, L3 ≈ 15 ns, RAM ≈ 80–100 ns, SSD ≈ 100 µs, HDD ≈ 10 ms.
• Cache line: Smallest unit of transfer. Reading one byte fetches 64 bytes.
• Locality: Iterating arrays is fast, linked lists jump around in memory.
• False sharing: Two threads write different variables in the same cache line and slow each other.
• Struct of arrays vs. array of structs: Layout decides cache efficiency.
• Prefetching: The CPU loads ahead.

See also: Caching strategies and cache invalidation, Amdahl’s law and scaling