Sharding
Partitioning rows across machines by a key. Hotspots and resharding are the cost.
Level: advanced
What is it?
Sharding distributes subsets of a dataset across different database nodes, usually by a partition key. Each shard owns some range or hash space, allowing storage and write capacity to scale horizontally when a single node is insufficient.
Why does it matter?
Sharding can increase capacity but makes cross-shard transactions, joins, uniqueness, rebalancing, and operational tooling more complex. The key determines distribution and can become a hotspot if traffic is skewed.
Mental model
Hash-based sharding spreads keys across partitions; range-based sharding preserves key ranges but can overload popular or newly appended ranges. A routing layer must know which shard owns each key.
How it works
- Choose a shard key that distributes writes and aligns with common queries.
- Hash partitioning often spreads load but makes range scans across key values harder.
- Range partitioning can make range queries efficient but may create hot partitions.
- Global uniqueness and transactions across shards require coordination or application-level design.
- Resharding requires a migration/copy strategy, cutover rules, and consistent routing during movement.
Steps
- A request arrives with customer_id.
- The routing layer computes or looks up the owning shard.
- The shard executes the local query, using local indexes and transactions.
- Cross-shard reads or updates coordinate several shards and must define partial-failure behavior.
Real-world usage
- Planning horizontal database scale-out.
- Designing tenant-based, range-based, or hash-based partitioning.
- Explaining why a hot shard limits overall capacity.
Common mistakes
- Choosing a timestamp key that sends all writes to one shard.
- Assuming one shard key suits every query pattern.
- Resharding without a plan for dual writes, cutover, verification, and rollback.
Interview questions
What makes a good shard key?
It distributes workload and data while aligning with common access patterns and required locality, uniqueness, and transaction boundaries.
Why is a hot shard a problem?
The busiest shard can saturate before others, so adding more total nodes does not improve capacity for the skewed workload.