Skip to main content
Consistent Hashinglesson 3 of 4 · 3 min read

Virtual Nodes

Random points do not spread evenly

The plain ring has a load problem the diagrams politely hide.

Your server positions come from hashing, which makes them random, and random points on a circle do not politely space themselves out. With 4 servers it is entirely normal for one to own 40 percent of the circle and another 12 percent.

What happens next is predictable. The overloaded one runs hot, gets blamed as a bad machine, gets replaced, and the replacement hashes to a different random spot and inherits a different unfair share. Worse, when a server dies its entire arc lands on one neighbour, doubling that machine's load at the exact moment your cluster is already hurting.

Many small arcs instead of one big one

Fix both with brute statistics. Instead of one position per server, give each of yours a hundred or so, by hashing its name with a counter appended.

Now your circle holds hundreds of points, and each server's share is the sum of many small random arcs. Sums of many random pieces cluster near their average, so shares land within a few percent of equal instead of tens of percent apart.

Failure behaviour improves for the same reason. A dead server's hundred arcs are scattered all round your circle, so its load spreads over every survivor in thin slices instead of crushing one neighbour.

Mixed hardware gets a clean knob out of it too: give the machine with twice the memory twice the virtual positions and it naturally owns twice the keys.

You pay in bookkeeping. Your sorted array grows by whatever factor you chose, lookups stay logarithmic, and in a storage system more ranges means more work during repairs and rebalancing.

Cassandra learned this the hard way. It shipped with 256 virtual positions per server, then cut the default to 16, having found that operational pain scales with the count as well. More smooths your load. Too many drowns you in ranges.

the shape of it
Server A40% of ringServer B12% of ringPlain ringWith 128 eachshares within a few %random spotrandom spotspread each server
step 1 of 2
One position per server lands unevenly; many small positions average out.

Worked example

Asha stands up a 4-node cache ring on Redis, an in-memory store, for a gaming backend, one position per node. Within a day, node 2 sits at 78 percent memory while node 4 idles at 30, and the on-call assumes a hot-key bug. It is not; a quick script summing arc lengths shows node 2 simply owns 41 percent of the circle by hash luck. She switches the client to 128 virtual nodes per server. The sorted array grows from 4 entries to 512, lookups stay around a microsecond, and ownership lands at 24 to 26 percent per node. Three weeks later node 3's host dies at 2am. Its 128 scattered arcs spill onto the other three nodes at about 8 percent extra load each, no single machine spikes, and the page resolves as "replaced instance, no user impact" instead of a cascading eviction storm.