Redis Deep Divelesson 4 of 4 · 3 min read

Replication, Sentinel, and Cluster

Two steps, often confused

One Redis process caps out at one machine's memory and roughly a hundred thousand operations a second. Growing past either limit takes two distinct steps, and conflating them is a common design error.

Replicate first, which copies the same data to more machines. Replicas connect to the primary and receive a stream of writes asynchronously.

Weigh that word asynchronously, because it is load-bearing. Your primary acknowledges a write before any replica has it, so losing the primary can lose acknowledged writes, and a read served by a replica can be milliseconds behind.

Take read capacity in exchange, and more importantly a warm standby.

Give failover a coordinator, and Sentinel is that. Its processes watch the primary, agree by majority that it is gone, promote a replica, and tell your clients the new address. Expect failover to take tens of seconds.

Splitting the data

Split the data second, with Cluster, once it no longer fits one primary. The keyspace divides into 16,384 slots, each node owns a range, and every key hashes to exactly one slot.

Let your clients learn the map and route each command to the right node, with a node that receives a misplaced key answering with the correct address.

Each shard carries its own replicas, so a cluster is really several replicated groups behind one keyspace.

Pay the visible cost of sharding. Commands touching several keys at once only work when those keys live in the same slot.

Force that with hash tags, where only the braced part of a key is hashed, pinning one user's keys together. Design your key names for this before you shard, not after.

Take the sizing advice that falls out. A primary with replicas and a coordinator covers most teams for years. Reach for Cluster when your working set outgrows one machine's memory or your writes outgrow one primary, and not before, because those multi-key restrictions are a real tax on your application code.

the shape of it

step 1 of 5

Replication copies one dataset; Cluster splits it. These are different steps.

Worked example

Oscar's social app runs one Redis primary with two replicas for its session and feed caches, 60 GB of data on 128 GB machines, comfortable. Two years of growth later the working set hits 200 GB and the eviction rate on hot feed data is hurting the hit rate. He migrates to Redis Cluster with 6 shards, each a primary and replica pair, about 35 GB per shard. The migration surfaces exactly one class of bug: a feed endpoint that used a single MGET across 30 users' profile keys, which now span shards and fail. The team applies hash tags so each user's own keys stay colocated, and rewrites the cross-user MGET as parallel per-shard fetches, adding about 1 ms. Failovers are now automatic per shard; a node loss in month three goes unnoticed by users entirely.

Redis Deep Dive: wrapping up

In the real world

01Twitter kept timelines in Redis for years, materializing each user's home timeline as an in-memory list that fan-out workers pushed tweet IDs into.
02GitHub uses Redis both as a cache and as the backing store for Sidekiq-style background job queues built on list push and blocking pop operations.
03Instagram published an early engineering post about cutting Redis memory dramatically by bucketing millions of key-value pairs into hashes, exploiting Redis's compact encoding for small hashes.
04Discord has described using Redis extensively for presence and session data, where sub-millisecond reads matter and the data is reconstructible on loss.
05Stack Overflow serves its question pages with a two-layer cache, local memory in front of Redis, and has run the whole site's Redis load on a remarkably small number of instances.

Questions people ask

If Redis is single-threaded, how does it handle thousands of clients at once?

Through I/O multiplexing. An event loop monitors all client sockets and processes commands one at a time as they become ready, and since each command takes microseconds, one thread interleaves thousands of clients without them noticing. Newer versions also offload network reads and writes to I/O threads, but commands still execute serially.

Can I use Redis as my primary database?

You can, and some teams do for specific datasets, but you are accepting that everything must fit in RAM, that default persistence can lose about a second of writes, and that replication is asynchronous. For data you cannot regenerate or afford to lose, a disk-based database with synchronous durability is the safer primary, with Redis as the speed layer in front.

When should I move from a single Redis instance to Redis Cluster?

When the dataset no longer fits in one machine's memory after eviction tuning, or write throughput exceeds what one primary sustains. If you need availability rather than capacity, replicas with Sentinel failover are much simpler. Cluster adds multi-key command restrictions and operational overhead, so treat it as the second step, not the first.

Quick review

Single-threaded command execution:: no locks needed. I/O multiplexing for concurrency
Data structures:: String, List (LPUSH/RPOP queue), Set, Sorted Set (ZSet), Hash, HyperLogLog, Stream
Sorted Set (ZSet):: ZADD key score member. ZRANGE for ranked leaderboards. O(log N) per op
Persistence:: RDB (point-in-time snapshot, compact) vs AOF (Append-Only File, every write logged, more durable)
Distributed lock:: SET key val NX EX 30. Atomic set-if-not-exists with TTL. Use Redlock for multi-node
Pub/Sub:: PUBLISH + SUBSCRIBE. Fire-and-forget, no persistence. Use Streams for durable messaging
Throughput:: ~100k ops/sec per instance (1M+ with pipelining or Redis Cluster). Latency: sub-millisecond. Bound memory with maxmemory + eviction policy

the trade-off

All data in RAM. Expensive at scale. Redis is not a primary database; it's a speed layer.

in the room

Cache layer, session store, rate limiter, real-time leaderboard, distributed lock, pub/sub fanout.

the short version

Replicas add read capacity and failover; Cluster shards the keyspace when one box is not enough.