Skip to main content
Circuit Breakerlesson 3 of 3 · 3 min read

Tuning and Fallbacks

Two ways to get it wrong

An untuned breaker fails in two directions, and you will meet both.

Too sensitive and it trips on noise. A routine deploy causes ten seconds of connection errors, your breaker opens, and now a perfectly healthy service is being refused for a minute, an outage the breaker itself manufactured.

Too tolerant and it never trips until your thread pools are already gone. The false-positive risk is real and it peaks during traffic spikes, when a brief latency blip looks identical to early sickness.

Tune from your measurements rather than from defaults. Set your call timeout from that dependency's observed p99 plus margin. A two second timeout on a service whose p99 is 200 milliseconds means waiting ten times longer than any healthy response before giving up.

Use rate-based thresholds over a minimum number of calls, so three failures out of five at quiet times do not trip anything. Count slow calls alongside failed ones. Keep the open period modest, 10 to 60 seconds, since your half-open probes make long punishments pointless.

Watch state transitions in your metrics, and alert on a breaker that stays open for minutes, because that means your dependency is genuinely down and a person should know.

The fallback is a product decision

Treat the fallback as a product decision wearing an engineering costume, and give it the same review a product decision would get.

Climb the ladder of options. Serve a cached previous answer, since stale recommendations beat none. Serve a static default, like an empty reviews section. Compute something simpler, like rule-based fraud scoring instead of the vendor's model. Queue the work for later, accepting the order and sending the receipt when email recovers. Or return an honest error for the things that cannot degrade, like taking the payment.

Choose per endpoint, deliberately. And test that fallback path under load, regularly, because the classic failure is a fallback reading from a cache that is empty precisely because the dependency has been down, discovered for the first time during the incident it was built for.

the shape of it
Call refusedBreaker openfails in microsecondsStale cacheStatic defaultHonest errorfor payments1. no network2a. last good answer2b. empty section2c. cannot degrade
step 1 of 2
The fallback is a product decision: which of these is acceptable differs per endpoint.

Worked example

An e-commerce team ships breakers on all outbound calls with library defaults: 5-failure trip count, no minimum volume, 60-second open. First Black Friday, at 00:03, a 15-second latency blip in the loyalty-points service (autoscaling catching up to the surge) trips its breaker on 5 slow calls. Points vanish from checkout for a full minute, and support gets 200 chats from confused customers, a mini-outage the breaker caused. Sameer's postmortem retunes it: trip at 50 percent failures over a minimum of 30 calls in 10 seconds, timeout set to 600 ms against the service's measured 180 ms p99, open for 15 seconds. The fallback changes too: instead of hiding points, checkout shows the last cached balance with an "updating" note. The next surge blips the same service for 12 seconds, and the breaker correctly stays closed while nobody notices anything.

Circuit Breaker: wrapping up

In the real world

  • 01Netflix built Hystrix to wrap every inter-service call with breakers, bulkheads, and fallbacks after cascading failures in its early microservices era; it was deprecated in 2018 in favor of Resilience4j and adaptive concurrency limits, but its design shaped the whole category.
  • 02Resilience4j (Java) and Polly (.NET) are the mainstream in-process implementations, offering rate-based thresholds, slow-call detection, and half-open probe configuration as library primitives.
  • 03Envoy and service meshes like Istio enforce breaker-adjacent protections at the proxy layer, capping connections and pending requests per upstream and ejecting hosts that keep returning 5xx (outlier detection), so polyglot fleets get protection without per-language libraries.
  • 04Michael Nygard's book Release It! introduced the circuit breaker to software in 2007, drawing on real trading and airline outages where one hung integration point took down entire systems.
  • 05AWS documents breakers alongside retries with exponential backoff and jitter in its builders' library, because unmanaged retry storms are one of the most common ways cloud systems amplify their own failures.

Questions people ask

How is a circuit breaker different from just setting a timeout?

A timeout bounds one call; a breaker acts on the pattern across many calls. With only a timeout, every request still waits out the full timeout against a sick dependency and still adds load to it. Once a breaker trips, requests fail in microseconds without touching the network, freeing your threads and giving the dependency room to recover. You want both: the timeout detects individual slow calls, the breaker responds to the trend.

Where should the circuit breaker live, in my code or in the infrastructure?

Both layers exist and complement each other. In-process libraries like Resilience4j and Polly give you per-call-site breakers with application-aware fallbacks, like serving a cached response. Proxy-layer protection in Envoy or a service mesh caps connections and ejects failing hosts uniformly across all services regardless of language. Large systems typically run mesh-level protection as a floor plus in-process breakers where fallback logic needs business context.

What should happen to requests when the breaker is open?

That is a per-endpoint product decision. Options in rough order of preference: serve cached or stale data, return a static default, compute a degraded answer locally, queue the operation to complete later, or fail fast with a clear error. The one wrong answer is silently pretending the operation succeeded. And test the fallback path regularly, because a fallback that only runs during disasters tends to be broken by the time one arrives.

Quick review

Closed state:
requests flow normally. Count failures. If error rate exceeds threshold → Open
Open state:
immediately reject all calls (fail fast). Return cached fallback or error. Start recovery timer
Half-Open state:
after timeout, allow limited trial requests. If they succeed → Closed. If fail → Open again
Prevents cascade failures:
service A calling B calling C, one failure cascades. Circuit breaker at each call boundary
Fallback strategies:
cached response, default value, degraded feature set, queue for retry
Libraries:
Hystrix (Netflix, deprecated), Resilience4j (Java), Polly (.NET), Envoy built-in
Metrics to watch:
error rate, latency percentiles, circuit state. Alert on prolonged Open state
the trade-off

False positives can trip circuit on healthy services during traffic spikes. Tune thresholds carefully.

in the room

Any synchronous call to an external or internal service that might fail or time out.