News Feed / Timelinelesson 4 of 4 · 3 min read

Ranking and Pagination

Pagination breaks first

Pagination breaks first in feed systems, so handle it first.

Understand why page numbers cannot work. Between page one and page two new posts arrive, everything shifts down, and your user sees the same items twice.

Use a cursor instead. Your first response includes an opaque marker encoding the position, in the simplest case the sortable identifier of the last item. The next request asks for 50 items older than that marker, which stays stable no matter how many posts arrived above it.

Notice the behaviour that gives you. New content appears only when somebody deliberately pulls to refresh, exactly how people expect a feed to work.

Ranking is a layer

Treat ranking as a layer rather than a rewrite. A chronological feed serves the sorted set directly. A ranked one inserts a scoring step between fetching candidates and responding.

Pull a few hundred candidates, call a scoring service that runs a model over things like how close you are to the author, what you engaged with before, how recent it is and what kind of post it is, then sort.

The first famous version of this was literally three terms multiplied together, before deep models replaced the formula. The architecture survives every model change: your candidates come from the same cache, and scoring is a stateless service you can scale and test independently.

Name the consistency wrinkle that ranking creates. Scores change between requests, so a freshly ranked second page might re-include something from the first.

Freeze the order for the session, as production systems do, making the cursor a snapshot held server-side for that scroll.

Accept staleness as the last honest trade. Precomputed feeds drift, so you unfollow somebody and their post is still sitting in your cached set.

Filter at hydration time against the current follow list, and let fan-out clean up lazily. Chasing perfect freshness synchronously would bring back every cost precomputation exists to avoid.

the shape of it

step 1 of 3

Ranking is a layer between the cached candidates and the page, not a rewrite.

Worked example

Sofia's team ships infinite scroll for a news app using page numbers: /feed?page=2. Reviews start mentioning the app shows me the same story three times. The mechanism is textbook: a user reads page one for 90 seconds, 12 new stories arrive, everything shifts 12 positions down, and page two overlaps 12 of the 20 items they just read. Duplicate rate in analytics: 18 percent of loaded items during peak news hours. She switches to cursor pagination, with the cursor carrying the last item's Snowflake-style ID, and the query becomes WHERE id < cursor LIMIT 20 against the feed cache. Duplicates drop to zero and one unexpected metric moves: average session depth rises 22 percent, because users who stopped scrolling at the first repeat, assuming they had reached the end, no longer hit one.

News Feed / Timeline: wrapping up

In the real world

01Twitter's home timeline has long used the hybrid model its engineers described publicly: fan-out on write into Redis timeline caches for most accounts, with high-follower accounts merged in at read time to avoid tens of millions of insertions per tweet.
02Facebook's News Feed started with the EdgeRank formula (affinity x edge weight x time decay) in 2009 and evolved into deep learning ranking over thousands of features, while the candidate-then-score architecture stayed recognizable.
03Instagram moved from a purely chronological feed to ranked in 2016, stating that users missed 70 percent of their feed under chronological ordering, and later reintroduced chronological as an option.
04Etsy and Slack both published write-ups on cursor pagination replacing offset pagination, for the same reason feeds need it: offsets shift under concurrent inserts and produce duplicates or gaps.
05LinkedIn's feed serves precomputed candidate sets through its FollowFeed system, which its engineering blog describes as timeline storage optimized for read-time merging across followed entities.

Questions people ask

When does fan-out on write stop making sense?

When follower counts make the write amplification unpayable or the wasted work dominates. A post to 50 million followers means 50 million cache insertions, mostly for users who will never scroll far enough to see it. Most systems set a follower threshold: below it, push at write time; above it, store once and merge at read time for the followers who actually show up.

Why store post IDs in the feed cache instead of full post content?

Three reasons: memory (8 bytes versus kilobytes per entry, across hundreds of entries for millions of users), consistency (edits and deletes happen in one place instead of a thousand cached copies), and cheap fan-out writes. The cost is a hydration step at read time, which stays fast because it is one batched fetch against a hot post cache.

How do feeds stay correct when users unfollow someone?

Lazily. The unfollowed account's posts remain in the precomputed feed cache until they age out or a cleanup pass removes them, and the serving layer filters hydrated results against the current follow list so the user never sees them. Synchronously rewriting feed caches on every unfollow would be fan-out on write for an event nobody is waiting on.

Quick review

Fan-out on write (push model):: when user posts, write to all followers' feed queues immediately. Fast reads, expensive writes
Fan-out on read (pull model):: compute feed at request time by fetching recent posts from followed accounts. Cheap writes, slow reads
Hybrid:: fan-out on write for regular users (< 10k followers). Fan-out on read for celebrities (> 1M followers). Twitter uses this
Feed storage:: Redis sorted set per user. Score = post timestamp. ZRANGE for paginated reads
Ranking:: chronological (simple) → engagement-based (likes × weight + recency × weight) → ML models (Twitter, Instagram)
Pagination:: cursor-based (timestamp or post_id) not page numbers. Avoids inconsistency as new posts arrive
Caching feed:: pre-generate top N items for active users. Lazy generate for inactive users (last seen > 7 days)

the trade-off

Pre-computed feeds go stale (user unfollows but still sees their posts until refresh). Freshness vs performance.

in the room

Tests fan-out trade-offs, the celebrity problem, feed ranking, and cache design for personalized data.

the short version

Cursors keep infinite scroll stable; ranking is a candidate-scoring layer on top of the same plumbing.