Skip to main content
Long Pollinglesson 3 of 3 · 2 min read

Long Polling at Scale

Connections are the currency

Held connections are the currency you are spending here.

Every parked request pins a socket, a file handle, and whatever state your framework keeps per request. Plan capacity by connections at once, not by requests a second.

On an event loop the cost per connection is small, a few tens of kilobytes, and one machine comfortably parks 50,000 requests. Raise your file handle limits before anything else, because the default of 1,024 on many Linux images is your first outage.

Solve fan-out second. With one server, waking a parked request is a lookup inside the process. With ten servers behind a balancer, the event for Anna might be created by a request that landed on server 3 while Anna's poll is parked on server 7.

Put a pub/sub backbone underneath. Every server subscribes to channels for the users it is currently holding, and whatever produces events publishes without caring who holds whom.

Take comfort that this is the same architecture server-sent events and WebSocket fleets need, so the work transfers if you migrate later.

Stampedes

Expect stampedes third, and expect them to page you. A deploy or a crash drops tens of thousands of parked requests at once, and every one of those clients reconnects immediately, each doing a handshake and an auth check.

Your balancer sends that herd at the surviving servers, which can take those down too.

Defend on three fronts. Drain servers slowly before restarting them. Add a few seconds of randomness to client reconnects. And make reconnect auth cheap, a token check instead of a database lookup.

Know your exit as well. When each client is getting close to a message a second, the overhead of reconnecting per message means you are running a worse version of server-sent events. Switch then, rather than tuning.

the shape of it
POST /sendlands on any boxRedis pub/subServer A18k parkedServer B17k parkedWaiting clientspublishcomplete requests
step 1 of 3
A pub/sub hop lets an event created on any server wake a request parked on whichever server holds it.

Worked example

A quiz app runs live game updates over long polling: 4 servers, about 45,000 parked requests at peak on a Sunday night. A routine deploy restarts servers one at a time with a 5 second pause between them, which drops roughly 11,000 clients per restart. Every one reconnects within a second, the auth middleware does a Postgres session lookup per request, and the database hits max connections, which fails health checks on the remaining servers and turns a rolling deploy into a 9 minute full outage. The postmortem produces three fixes: reconnect jitter of 0 to 4 seconds in the client, session tokens validated from Redis, an in-memory store, instead of the main database, and a 60 second drain per server during deploys. The next Sunday deploy goes completely unnoticed, which is the point.

Long Polling: wrapping up

In the real world

  • 01Dropbox desktop clients learn about folder changes through list_folder/longpoll, an HTTPS long poll whose timeout parameter accepts 30 to 480 seconds, then fetch the actual deltas separately.
  • 02Amazon SQS long polling holds ReceiveMessage for up to 20 seconds via WaitTimeSeconds, and AWS's own docs recommend it over short polling to cut empty receives and cost.
  • 03Telegram's Bot API offers getUpdates with a timeout parameter as its official long polling mode, the standard alternative to webhooks for bot developers.
  • 04Facebook Chat launched in 2008 on Comet-style long polling, with Erlang channel servers whose whole job was holding open requests for online users.
  • 05Gmail's in-browser chat ran on Google's BrowserChannel, a long polling transport built years before WebSocket existed in any browser.

Questions people ask

What timeout should the server use for a long poll?

Below the shortest idle timeout of anything between the client and your server. AWS ALBs default to 60 seconds and Nginx's proxy_read_timeout defaults to 60, so 25 to 30 seconds is the common safe choice. On timeout, return an empty 204 and let the client immediately reconnect; the timeout response doubles as proof the connection is still alive end to end.

How is long polling different from SSE?

A long poll delivers one response per request, then the client reconnects, while SSE keeps a single response open and streams many events down it. SSE also ships reconnection and resume via Last-Event-ID in the browser's EventSource API, which long polling makes you build yourself with cursors. If the server can push and traffic is one-way, SSE is usually less code for a better result.

Do I lose messages between polls?

You will unless you design against it. Events that fire in the 50 to 200 ms gap between one response and the next request have no parked request to ride. Carry a cursor: each response includes the last delivered event id, the client echoes it back as since=<id>, and the server replays anything newer before parking again.

Quick review

Client sends request; server does NOT respond until new data is available (or timeout ~30 to 60s)
On response:
client immediately sends next long-poll request. Maintains near-continuous connection
Eliminates empty responses vs short polling. Approach used by early chat apps (Facebook Chat circa 2008)
Server must hold many open connections. Needs async/non-blocking I/O (Node.js, Nginx, async Python)
HTTP connection timeout requires reconnect, each reconnect has TCP + TLS overhead
Not suitable for high-frequency updates (>1/sec). WebSocket is better
Still client-initiated HTTP. Works through any proxy, firewall, or CDN without configuration
the trade-off

Open connections consume server memory. Can't handle high-frequency bidirectional communication efficiently.

in the room

Real-time updates needed but WebSocket infra isn't available. Medium frequency events (< 1/sec). Legacy compatibility.