Why is a bloom filter acceptable for URL dedup if it has false positives?

Because the failure is bounded and cheap: a false positive means one URL is wrongly considered seen and never crawled, roughly 1 percent of candidates at typical sizing. For broad crawls that miss rate is invisible in coverage terms. Crawls that need exactness pair the bloom filter with an exact store, using the filter to eliminate the vast majority of lookups before touching disk.

How does a crawler avoid infinite crawler traps?

With layered limits rather than one clever detector: maximum URL depth and length, a per-domain page budget, URL normalization that strips session IDs and boils away infinite query variants, and content fingerprinting that notices when a domain keeps serving near-identical pages. Traps still get partially crawled; the caps ensure they waste a bounded slice of the budget.

What actually limits crawl speed, the crawler or the web?

Per-domain politeness, almost always. Aggregate fetch capacity is easy to add with more async workers, but a domain crawled at 1 request per second yields at most 86,400 pages a day no matter how large your fleet is. That is why coverage math has to be done per domain, and why big sites get supplementary channels like sitemaps and change feeds.

Common Designschapter 4 of 8 · 4 lessons

Web Crawler

Systematically download and index the web at scale. Breadth-first, politeness-aware.

← All chapters View in cheatsheet

The program is four lines. Fetch a page, pull out the links, add them to a list, repeat.

Run it and within a minute you have hammered one site with a thousand requests and been banned, followed a calendar that generates a new page for every day until the year 3000, downloaded the same article at nine different addresses, and filled your disk with a video you never wanted.

None of that is a bug in the four lines. The web is a graph full of loops and mirrors, some of it is deliberately hostile to programs like yours, and the polite conventions are not enforced by anything except you.

Every real crawler is a distributed system wrapped around that loop to stop it destroying both its target and itself, and at a billion pages a month the constraint is not speed. It is bandwidth, storage, and not getting banned.

Lessons

4 in this chapter