At what point do I actually need to shard?

When sustained writes approach what one primary can handle, roughly 5,000 to 10,000 per second for a tuned relational database, or when the working set has outgrown what RAM and archiving can contain, and you have already used caching, read replicas, and a bigger instance. Measure and project first; sharding a database doing 100 writes per second is pure cost.

Can I change my shard key later?

Only by migrating everything, since the key determines where every row lives; changing it is a full resharding project with dual writes and backfills. That is why you choose it against your actual query log. For specific queries the key serves badly, a secondary copy of the data organized by a different key is far cheaper than re-keying the world.

Do unique constraints and foreign keys work across shards?

No. Each shard enforces constraints only on its own rows, so a globally unique email needs either a dedicated lookup service or a design where all rows for that scope live on one shard. Globally unique IDs are usually solved with generation schemes like Snowflake IDs or Instagram-style IDs that embed the shard.

Scalingchapter 4 of 4 · 4 lessons

Database Sharding

Horizontally partition data across multiple DB instances by shard key.

← All chapters View in cheatsheet

One database machine, well tuned, takes somewhere around 10,000 writes a second before it stops keeping up. You are doing 9,000.

Read copies do not help, because every write still lands on the one leader. Caching does not help, for the same reason. A bigger machine buys you one more step, and there is no step after the biggest machine your cloud sells.

Sharding splits the rows across separate databases so ten machines take ten times the writes. It is the answer, and it is also a migration measured in quarters that complicates every feature you build afterwards, which is why the rest of this chapter is mostly about waiting as long as you possibly can.

Lessons

4 in this chapter

Why Shard, and Why WaitReplicas scale reads. Nothing scales writes on one box forever. Shard late, on purpose.2 min
Picking a Shard KeySpread the load evenly and keep common queries on one shard. Those two goals fight.3 min
Resharding Painhash mod N moves everything when N changes. Logical shards make it a move, not a rehash.3 min
Living with Cross-Shard QueriesQueries that name their shard stay fast. Everything else needs a plan, not a hope.3 min