Message Flow Across Servers

The connection

Start with the connection protocol, and the modern answer is a WebSocket: a persistent two-way connection your server can push down at any moment.

Reject the alternatives for good reasons. Polling every few seconds burns battery and adds seconds of latency. Holding a request open is a workaround from before sockets existed. Keep both in your pocket for hostile corporate proxies, and design around the socket.

Trace what happens on connect. Your client reaches one of many chat servers through a balancer, authenticates, and that server records where it lives, so your registry now says this person is on server 7.

The routing problem

Meet the routing problem immediately. Asha sends a message to Ben. Asha's socket terminates on server 3 and Ben's on server 9, and server 3 cannot hand the message to Ben, because it does not hold his socket.

Put a broker between your chat servers, the standard answer. Server 3 saves the message and then publishes it to a channel named for Ben.

Server 9 subscribed to channels for everybody it holds when they connected, so it receives that publish and writes the message down Ben's socket. Neither server knows the other exists, and the broker is the meeting point.

Get the sequence right, because interviewers probe it. Save first, then publish.

See what goes wrong the other way round. Publish before the write commits, then crash, and Ben saw a message that does not exist in history and never reaches Asha's other devices. Saving first means your worst crash outcome is a delayed delivery that offline sync repairs.

Acknowledge back to Asha only after the save, too. That acknowledgement is what her client draws as the single sent tick.

void send(Message m) {
  // Save before publishing. Do it the other way round and a crash
  // between the two means the recipient saw a message that does not
  // exist in history and never reaches the sender's other devices.
  store.append(m.conversationId(), m);

  // Asha is on server 3, Ben on server 9. Server 3 does not hold
  // Ben's socket, so it publishes to a channel named for Ben and
  // whichever server holds him picks it up.
  broker.publish("user:" + m.recipientId(), m);

  // Only now does the sender's client draw the single tick.
  ack(m.senderId(), m.id());
}

// On the receiving server, subscribed for everyone it holds:
broker.subscribe("user:" + userId, m -> sockets.get(userId).write(m));

Worked example

Diego's team at a logistics startup builds driver-dispatcher chat with two WebSocket servers behind a round-robin load balancer, and delivery works in every test. In production, drivers report messages arriving only sometimes, roughly half. The bug is exactly the two-server problem: when a dispatcher on server A messages a driver on server B, server A looks in its local in-memory session map, finds nothing, and quietly stores the message as offline. Half of all pairs land on the same server by chance, which is why it half-works. The fix takes a week: a session registry in Redis, an in-memory store, mapping user to server, and Redis Pub/Sub so each server subscribes to its own inbox channel. Cross-server delivery goes from 50 percent to 99.9, and the remaining failures turn out to be real disconnects.