Software Engineering Glossary

Web Crawler

Also known as: Crawler Web spider Search engine crawler

A web crawler discovers URLs and downloads the documents behind them, starting from seeds and from the links it finds along the way. Its job is a fresh, polite, deduplicated corpus of pages, not search results. Ranking and querying happen later, in a separate index. At any serious size, the design is dominated by the URL frontier, per host politeness, and duplicate detection rather than by the HTTP fetch itself.

Key Takeaways

  • A crawler builds a corpus. A scraper extracts fields. A search index answers queries. They are three different systems.
  • Politeness is a correctness rule. One host must not receive a burst of fetches just because you added workers.
  • Shard by hostname so robots.txt, the delay, and the queue for a host all live on one machine.
  • Seen URLs and stored pages have to survive restarts, or the next deploy recrawls the world.

How It Works

  1. Seed URLs and sitemaps enter a frontier that orders work by priority and by host.
  2. A fetcher downloads a URL only when robots.txt allows it and the host’s delay has elapsed.
  3. The parser extracts links, canonicalizes them, and drops ones the seen set already contains.
  4. Page bodies go to object storage. A queue notifies the indexer. The URL is scheduled again if it should stay fresh.

Where It Is Used

  • Googlebot crawls the public web under a crawl budget per hostname, and publishes how it reads robots.txt.
  • Common Crawl stores fetches as WARC files that anyone can process later.
  • Mercator, the classic research crawler, separated priority from per host politeness inside the URL frontier.

Related glossary terms