Ten gateways, one limit
One server enforcing a limit is a counter in memory. The trouble starts when your limit is 100 a minute and ten gateway instances each see a slice of the traffic.
Count locally and a client spraying requests across all ten gets up to ten times your limit. The count has to live somewhere shared.
Reach for an in-memory store, the standard answer. Each gateway checks a key for that client on every request.
Watch for the subtle bug in the obvious implementation. Read, compare, then increment is a race, because two gateways can both read 99 at the same instant and both admit request 100.
Make checking and incrementing one atomic operation instead, which these stores let you do with a small script: read the bucket, refill it by elapsed time, spend a token, and answer yes or no, all as one step. Sub-millisecond, and one node handles on the order of 100,000 of them a second.
Pull one of two levers if that round trip on every request bothers you. Keep a bucket in each gateway's memory and reconcile in the background, trading a small enforcement error for zero added latency on the hot path.
Or accept the millisecond, which for most APIs is noise against a 50 millisecond request.
Fail open or fail closed
Design the failure mode on purpose, because your store will have a bad day. Fail open and losing the limiter means no limits, acceptable for product APIs where availability wins.
Fail closed and losing the limiter rejects everything, correct for login endpoints and payment attempts where the limit is a security control. Most teams fail open with an alarm, plus a coarse local fallback so a total loss still leaves some ceiling.
Keep this state away from your cache, too. An eviction under memory pressure should never silently reset everybody's buckets.
Worked example
Ravi's team runs 12 Envoy gateway pods with per-pod in-memory limits of 100 requests per minute per key, believing that enforces 100 total. A penetration test report shows a single key achieving 1,140 requests per minute by round-robining connections across pods. The fix is a Redis-backed token bucket via a Lua script; measured cost is 0.6 ms added at p50, invisible against their 80 ms median API call. They configure fail-open with a local 200 per minute fallback per pod and an alert. Four months later that decision gets exercised: Redis fails over and the limiter degrades for 40 seconds. During the window, the fallback caps worst-case leakage at about 2,400 per minute per key, nothing falls over, and the incident review takes ten minutes.