What is the crawl for
Ask what the crawl is for before you design anything, because its purpose sets every parameter.
A search crawl wants breadth and freshness across billions of pages. A price monitor wants a fixed set of product pages refetched hourly. An archive wants depth and completeness.
Assume the search version for an interview: a billion pages a month, content stored for an indexing pipeline downstream, robots files respected, and nobody's website taken down.
Do the throughput arithmetic. A billion pages over 30 days of seconds is about 385 pages a second sustained, and that is not a large request rate. One good machine can issue it.
Find the constraint elsewhere. At an average page weight of 500 kilobytes, 385 pages a second is roughly 190 megabytes a second coming in, and 500 terabytes of raw markup a month. Bandwidth and storage are your budget lines, not processors.
Look at the subtler resource: concurrency. A page fetch takes maybe half a second, dominated by the remote server. So 385 a second means about 200 fetches in flight at any moment, and far more once slow sites drag your tail out.
Recognise the workload: thousands of open connections and almost no computation per connection.
The non-obvious requirements
Name the non-obvious requirements before you draw the diagram. Politeness is hard, meaning per-domain rate limits and honouring the robots file, or your crawl dies by getting banned.
Treat deduplication as correctness rather than optimisation. The web is a graph full of cycles and mirrors, and without a seen-before check your crawler revisits the same pages forever.
Code defensively, because the web is adversarial. Traps generate infinite spaces of URLs, and some servers happily return success with garbage forever.
Worked example
Andre plans a crawl of 40 million e-commerce product pages for a price-comparison startup, refreshed daily. Sustained rate: 40M / 86,400, about 460 fetches per second. Then he segments by domain and finds the catch: 60 percent of those pages live on just 20 retailer domains. Amazon's share alone is 8 million pages, and at a polite 1 request per second per domain, 8 million sequential fetches take 92 days, not one. The daily-refresh requirement is arithmetically impossible under politeness, and no amount of workers fixes it, because the bottleneck is the per-domain limit, not capacity. He goes back to product with the math, and they renegotiate: top sellers refresh daily via retailer APIs and sitemap diffs, the long tail refreshes weekly. Estimation killed a doomed architecture before anyone built it.