Adding a shard, the naive way
Sooner or later you add shards, and the obvious setup makes that a catastrophe.
Say a row's home is the hash of its key divided by the number of shards. Going from 8 shards to 10 changes that answer for nearly every key you have. Almost all your data must physically move, while serving live traffic, without losing a write. Teams have frozen feature work for entire quarters over exactly this.
Two defences, installed on day one
Install one of two defences on day one, before you have any data to move.
Consistent hashing arranges shards in a ring, so adding one moves only the keys in its slice, roughly one shard's worth instead of everything. It is the same arrangement that keeps cache clusters stable when a machine joins.
The other, more common for databases, is to create far more shards than you have machines. Say 480 logical shards living on 32 physical ones, with a small table recording who holds what. Now growing means move logical shards 160 to 239 onto the new machine and update the table, which copies whole units intact and touches no individual row.
Even with good foundations the shape is fixed. Write to both the old home and the new one, copy the history across in the background, and compare checksums until the two agree. Then move reads over behind a flag, watch it, and only after that stop writing twice and delete the old copy.
Every step needs a way back, because the moment you move reads is when you discover the query nobody wrote down.
Decide your logical shard count once, generously, before a single byte of production data exists. Take your wildest estimate and round it up hard. Logical shards cost almost nothing, and the difference between 16 and 512 of them is invisible at launch and enormous in year four.
Worked example
Notion's engineers wrote up both halves of this story. In 2021 they escaped an overwhelmed Postgres install, a relational database holding the entire product, by sharding on workspace ID, deliberately creating 480 logical shards spread across just 32 physical machines, a number picked for how many ways it divides. In 2023 growth presented the bill and the insurance paid out: to triple capacity they moved to 96 machines, each now hosting 5 logical shards instead of 15, by syncing whole logical shards with replication streams, verifying with dark reads, and cutting over in stages. No per-row rehashing, no schema surgery, no downtime users noticed. The alternate timeline, hash mod 32 baked into the application in 2021, would have meant remapping nearly every row in the company under load.