Client and In-Process Cacheslesson 2 of 2 · 2 min read

In-Process Caching

A lookup that costs nothing

Inside your own process, a lookup costs nothing at all. No serialisation, no socket, no waiting. A read from a map in memory takes nanoseconds where a call to a shared cache takes tenths of a millisecond, roughly a thousand times the difference.

That gap is only worth having for data you fetch constantly. A feature flag read on every request, a currency table, a small lookup nobody edits. For anything larger the memory belongs to your application, and taking a gigabyte for cached data is a gigabyte your request handling no longer has.

Ten servers, ten caches

Now the part that catches people. Run ten servers and you have ten caches, and none of them knows about the others. Change a flag and the server you deployed to picks it up while the other nine carry on with the old value until their own copies expire. There is no delete that fixes all of them, because there is nothing they share.

The expiry is your tolerance for disagreement between your own servers. Thirty seconds means ten servers can hold different answers for thirty seconds. Fine for a feature flag. Not fine for anything a customer is paying for.

A real cache library beats a plain map. Caffeine and Guava give you a size limit, expiry, and eviction; a HashMap gives you an unbounded structure that grows until the process dies. That failure arrives in production, weeks later, as a memory problem nobody connects to the cache added in a hurry.

the shape of it

Each server holds its own copy, so a change is visible on one and invisible on the rest until their expiries run out.

bounded, expiring, and local

Java

// Caffeine, not a HashMap. The size limit and the expiry are the
// difference between a cache and a memory leak.
private final LoadingCache<String, Flag> flags = Caffeine.newBuilder()
    .maximumSize(1_000)
    .expireAfterWrite(Duration.ofSeconds(30))   // how long servers
    .build(db::loadFlag);                       // may disagree

Flag get(String name) {
  return flags.get(name);       // nanoseconds, no network at all
}

// There is no invalidate that reaches the other servers. This
// clears one process only, which is the whole trade.
void clearLocally(String name) {
  flags.invalidate(name);
}

Worked example

Arjun cached his forty row currency table in each server's memory with a ten minute expiry. Forty rows, read on every checkout, changed once a day. It removed a Redis call from the hottest path in the system and nobody noticed it had happened, which is the correct outcome.

The same trick failed the following quarter. A colleague cached feature flags the same way, then used one to disable a broken payment provider during an incident. The flag flipped, one server obeyed immediately, and eleven others kept routing payments to the broken provider for the remaining nine minutes of the expiry.

The flags moved to Redis that week. Ten minutes of disagreement is nothing for an exchange rate and a long time during an outage.

Client and In-Process Caches: wrapping up

In the real world

01Stack Overflow serves a large share of its traffic from in-process caches on the web servers themselves, on the reasoning that the fastest call is the one that never leaves the machine.
02Netflix ships configuration to services through a library that keeps values in process and refreshes them in the background, so a lookup is a field read rather than a network call.
03Browsers cache aggressively by default, which is why front-end builds put a content hash in every filename: it makes a changed file a different file rather than a correction nobody can deliver.
04Redis clients keep cluster topology in process, so an operation goes straight to the node holding the key instead of asking the cluster where it lives every time.

Questions people ask

When is in-process caching worth it over Redis?

When the data is small, read constantly, and can be a little out of date on some servers and not others. Feature flags, configuration and small lookup tables qualify. Anything a user would notice being inconsistent between two page loads does not.

How do I invalidate an in-process cache across servers?

You cannot do it directly, which is the point. The usual approach is a short expiry so every server converges within a known window, or a pub/sub message that tells each process to drop its own copy. Once you are running the second one you have most of the complexity of a shared cache and might as well use one.

Why not just use a HashMap?

Because it has no size limit and no expiry, so it grows until the process runs out of memory, and it never notices that a value has gone stale. Caffeine gives you both in one builder, and the failure it prevents shows up weeks later as an out-of-memory error nobody connects to the cache.

Quick review

Client-side:: the browser or app keeps the copy, so a hit costs no network at all and you cannot reach it to fix it; Cache-Control max-age is a promise you cannot break, which is why versioned filenames beat trying to invalidate; ETag trades a round trip for the body, which makes a page cheap rather than fast
In-process:: a map in your own memory, roughly a thousand times faster than a Redis call; Every server holds its own copy, so a change lands on one and the rest stay stale until their expiry; Use a bounded cache library rather than a HashMap, or the structure grows until the process dies

the trade-off

Both are faster than a shared cache because no network is involved, and both give up the thing a shared cache is for: one delete correcting every reader at once. The expiry you pick is the length of time your own servers are allowed to disagree.

in the room

Small, hot, rarely changing data where servers disagreeing for a few seconds costs nothing. Feature flags, configuration, lookup tables.

the short version

A map in your own memory, faster than Redis, and wrong on nine servers out of ten.