Sharding and partitioning

In short: With sharding a large data set is split into parts (shards) that sit on different servers. Storage and throughput then grow with the number of servers.

In more detail: A partition key decides which row goes into which shard. Good keys spread load evenly and keep related data together. Queries across several shards are expensive.

In Depth

• Range partitioning: For example A–F on server 1. Good for range queries, risk of hotspots.
• Hash partitioning: Hash of the key decides the shard. Even, but range queries are hard.
• Consistent hashing: When adding servers only a small part of the data moves.
• Rebalancing: Moving shards when servers are added.
• Hot keys: One very popular key overloads a shard.
• Joins and transactions: Costly across shards (see saga, 2PC).

See also: Consistent hashing, Distributed transactions: two-phase commit and saga