Two levels of deduplication
Run deduplication at two levels, and do not conflate them, because that is a common interview stumble.
One asks whether you have queued this address before, and it guards your frontier. The other asks whether you have stored these bytes before, and it guards your store, catching mirrors, session variants and the same article at ten addresses.
Treat address deduplication at a billion addresses as a memory problem. A set of a billion of them at sixty-odd bytes each is over 60 gigabytes and growing.
Reach for a bloom filter, the standard answer. At ten bits per element, a billion addresses fit in about 1.2 gigabytes of memory with a false positive rate near one percent.
Read the trade explicitly. A false positive means your crawler believes it saw an address it never saw, so that page is skipped forever.
For a search crawl, where silently missing one candidate in a hundred is fine. If it is not fine, back the filter with an exact store and let the filter reject cheaply first, keeping 99 percent of your lookups off the disk.
Content deduplication
Deduplicate content by hashing, at increasing sophistication. Exact duplicates fall to a hash of the body kept in a fingerprint store.
Catch near-duplicates, the same article with a different sidebar and timestamp, with similarity hashing. One well-known pipeline used a scheme mapping similar documents to fingerprints differing in only a few bits, so finding near-duplicates becomes a distance lookup.
Relax about storage, the calm part here. Raw markup is immutable blob data, so write it to object storage compressed, since markup squeezes five to one or better. Bundle it into the archival format the public web archives publish.
Put your metadata somewhere queryable: the address, the fetch time, the status, the content hash, the outgoing links. Deciding when to fetch a page again is tomorrow's problem, and it runs on that table.
Worked example
Common Crawl, the nonprofit whose dataset trains half the large language models in existence, shows the storage math at full scale. Each monthly crawl fetches on the order of 3 billion pages and publishes them as WARC files on S3, with each crawl adding roughly 90 TB compressed, several hundred terabytes raw. Their published stats also make the dedup case: crawl reports regularly show a meaningful fraction of fetched URLs resolving to duplicate or near-duplicate content, which is why fingerprints ship alongside the archives. A team at a startup fine-tuning models learned the lesson downstream: training on two months of Common Crawl without dedup meant the same boilerplate pages appeared thousands of times, skewing the model until they ran fingerprint-based filtering, the same simhash-family technique the crawl side uses.