Skip to main content
Consistent Hashinglesson 1 of 4 · 3 min read

The Resharding Problem

Hash and take the remainder

Spread your keys over four servers by hashing each key and taking the remainder. You will have this idea first, and it works.

The hash spreads your keys evenly, every client works out the same answer without talking to anyone, and a lookup costs you one piece of arithmetic. The failure is hiding inside the remainder you took.

Go from 4 servers to 5, and about 80 percent of your keys get a different answer than they did before. The pattern generalises: changing the server count remaps nearly all of the keys, so growing a pool of 10 to 11 moves roughly 90 percent of them.

Number carefully. Not the keys on one server. Nearly all of them, everywhere, in the same instant.

What it costs you

Work out the cost from what your servers hold. If they are caches, every remapped key is now a miss, so adding capacity to a cache pool drops its hit rate from 95 percent to nearly zero.

The full read load then lands on the database the cache existed to protect. That database, sized for 5 percent of traffic, takes 100 percent of it. Adding capacity took the site down, which reads like a joke until you have lived through it.

If they hold data rather than copies, the remapped keys must physically move to their new owner. Reshuffling 90 percent of 20 terabytes is days of network transfer and degraded service, so operators avoid resizing, so clusters run either wastefully over-provisioned or dangerously full.

State what you actually want, and the answer designs itself. You want a way of placing keys where changing the server count moves only the keys that have to move, roughly one server's worth. Everything else stays exactly where it is.

the shape of it
user:8642hash = 138% 4 serversbefore scalingServer 2entry lives here% 5 serversone node addedServer 3empty: missDatabaseabsorbs the misses138 % 4 = 2hit138 % 5 = 3missfetch from origin
step 1 of 3
Changing the divisor remaps most keys at once, and every remapped cache key becomes a database read.

Worked example

Jonas runs an 8-node memcached fleet in front of MySQL at a news site, mod-8 hashing in the client library, 96 percent hit rate. Ahead of an election night he adds 2 nodes for headroom and deploys the config at 7pm. Instantly hash % 10 disagrees with hash % 8 on about 90 percent of keys: hit rate craters to 9 percent, and MySQL, which had been seeing 400 queries per second, gets 9,000. Connections pile up, the site browns out for 25 minutes while the cache slowly rewarms, and the postmortem's first line is "adding cache capacity caused the outage." The fix that ships the next week is ketama-style consistent hashing in the client, and the test that proves it: adding an 11th node moves 9 percent of keys, and the hit rate dips 4 points for a few minutes instead of collapsing.