Software Engineering Glossary

URL Frontier

Also known as: Crawl frontier URL queue

The URL frontier is the set of URLs a web crawler has discovered and not yet fetched, including URLs waiting for a later recrawl. It has to do two jobs that fight each other: fetch important pages first, and avoid sending many requests to the same host at once. The standard design uses front queues bucketed by priority and back queues bucketed by host, with a clock that says when each host may be contacted again.

Key Takeaways

  • A single FIFO queue visits URLs in discovery order and will pile onto whichever host you happen to find first.
  • Front queues express priority. Back queues express politeness. A router moves URLs from one to the other.
  • One back queue per host, owned by one worker, makes the delay a local variable instead of a distributed lock.
  • Most of the frontier lives on disk. Only the heads of the queues need to sit in memory.

How It Works

  1. A discovered URL is scored and appended to the front queue for its priority.
  2. A router drains front queues, preferring higher priorities, and appends each URL to the back queue for its host.
  3. The scheduler pops a URL only from a back queue whose next-allowed time has arrived.
  4. After the fetch, that host’s next-allowed time moves forward by the politeness delay.

Where It Is Used

  • Mercator gave each worker thread one FIFO subqueue and placed URLs by canonical hostname, so one thread at most talked to a given server.
  • A disk frontier with in-memory buffers is how crawlers hold hundreds of millions of pending URLs without a RAM-sized queue.
  • Recrawl is the same frontier: a URL re-enters a front queue when its next fetch time arrives.

Related glossary terms