Density and Hotspots

Even cells, uneven world

Sharding by geohash prefix keeps each query on a single shard, the property you want. It also guarantees uneven shards, because businesses cluster where people do.

Do the comparison. A cell over midtown Manhattan holds a hundred times the rows of a same-sized cell in rural Nevada, so a fixed prefix length gives you shards differing by two orders of magnitude in both size and traffic.

Watch for the symptom, and it is not a crash. It is one shard 90 percent busy while its neighbours idle at 4 percent.

Resist the instinct to add shards, because it fails. A uniform split hands your new shards the same uneven distribution.

Stop treating prefix length as global instead. Assign shards by prefix ranges, and let a range split when its row count or its request rate crosses a threshold, exactly as a range-sharded database splits a hot region.

See the result: dense areas end up owned by longer, more specific prefixes, and empty regions stay on short ones. A routing table maps ranges to shards, small enough to cache everywhere and changing rarely.

The hotspot made of time

Meet a second kind of hotspot, and this one is about time rather than place. A stadium holding 60,000 people generates searches from one cell for three hours and nothing for the rest of the week, so provisioning for it permanently is waste.

Handle that with cache instead of shards. The candidate list for that cell is identical for every one of those people, so a single warm entry absorbs the whole event.

This is the case where caching by cell rather than by user pays for itself in one afternoon.

Take the general lesson into your interview. Any sharding key that comes from the physical world inherits the real world's skew, and your design has to answer for it instead of assuming things spread evenly.

// Fixed prefix length means Manhattan and rural Nevada get cells
// of the same size and wildly different row counts. A uniform split
// hands the new shards the same uneven distribution.

// Assign by prefix RANGE instead, and split a range when it gets hot.
record ShardRange(String fromPrefix, String toPrefix, int shard) {}

void maybeSplit(ShardRange r) {
  if (r.rowCount() > SPLIT_ROWS || r.requestsPerSec() > SPLIT_RPS) {
    split(r);        // dense areas end up on longer, more specific prefixes
  }
}

// The other hotspot is about time, not place: a stadium generates
// searches from one cell for three hours a week. Do not shard for
// that. Every one of those users gets the identical candidate list,
// so one warm cache entry absorbs the whole event.
cache.set("cell:" + cell, candidates, Duration.ofMinutes(5));

Worked example

A team shards their index on 5-character geohash prefixes and it looks balanced in staging, because their test data was generated uniformly. In production, the shard holding central London runs at 88 percent CPU while the shard covering most of Wales sits at 3 percent, and p99 on London searches is 340 ms against 18 ms elsewhere. Adding two shards does nothing, since the split is still uniform. The fix is a routing table with splittable ranges: London's prefix range splits three times, down to 7-character granularity, and spreads across 4 shards, while Wales keeps a single 4-character range. Peak CPU across the fleet lands between 40 and 55 percent and London p99 drops to 24 ms, on the same total hardware.