A loop drawn as a pipeline
Draw the architecture as a loop shaped like a pipeline.
Seed addresses go into your frontier, the queue of what to fetch next. Fetchers pull from it, resolve the name, download the page, and hand the raw bytes to a processor.
That processor checks for duplicate content and stores the markup in object storage. Then it parses out the links, normalises them, filters them against what you have already seen, and feeds the survivors back into the frontier.
Every arrow is a queue, which means each stage scales and fails on its own. A parser bug does not stop your fetching, and a slow day for storage does not stall your frontier.
Two stages hide the engineering
Look closely at two stages, because they hide real engineering. Resolving names at 400 fetches a second across millions of domains melts a naive setup. Every fetch needs a lookup, and the default resolver does them one at a time.
Run your own caching resolver and resolve names ahead of time for addresses nearing the front of the queue.
Keep your fetchers dumb and asynchronous. Thousands of concurrent connections per machine, strict timeouts, and size caps, because a two gigabyte response is an attack rather than a page. Check the content type before you download video you never wanted.
Normalise your extracted links before deduplication has any chance of working. Lowercase the host, strip the fragment, resolve relative paths, and strip tracking parameters.
Skip that and one page counts as three different addresses, and your frontier fills up with ghosts.
Keep the loop honest with backpressure. Parsing produces links faster than fetching consumes them, sometimes fifty new addresses per page, so your frontier only ever grows.
Put an admission policy at the gate: priority scoring, per-domain caps, depth limits. Without one, that queue becomes an unbounded to-do list for the entire web.
Worked example
Yuki builds a crawler for an academic search project and wires the whole loop as one Python process: fetch, parse, insert links, repeat. It works on 10,000 pages, then production behavior arrives. A university calendar site generates a next month link forever, and by morning the frontier holds 4 million URLs from one domain, all calendar pages for dates in the year 3000. Fetching stalls too: one department's server takes 45 seconds per response and her synchronous loop waits politely each time, dropping throughput to 2 pages per second. The rebuild splits stages into queue-connected services: async fetchers with 10 second timeouts, a parser tier, and a frontier gate enforcing max depth 12 and 10,000 URLs per domain. The calendar trap caps out and the slow server ties up one connection instead of the whole crawler.