2 / 6 Data & Analytics Pipelines

Crawl Politely

Design a web crawler that keeps a search index supplied with fresh copies of the public web.

Requirements

Functional

  • Fetch pages from operator-supplied seeds and from links discovered while crawling.
  • Obey standard robots.txt allow and disallow rules. As this crawler's own policy, also honor the nonstandard crawl-delay directive when a site advertises one, and never take a small origin down.
  • Detect exact and near-duplicate pages so the index stores one copy of content that appears under many URLs.
  • Prioritize recrawl: a news homepage matters more than a decade-old forum thread.
  • Emit fetched documents to an indexing pipeline.

Non-functional

  • Politeness is a hard constraint, throughput is a soft one.
  • No single slow or hostile host may stall the crawl.
  • A fetcher machine dying must not lose queued work.

Scale

  • 20B known URLs across 250M registrable domains.
  • 1B pages fetched per day: 1e9 / 86,400 is about 11.6k fetches/s average, plan for 20k/s peak.
  • Median page 110 KB of HTML, about 30 KB gzipped on the wire.
  • About 40 outlinks per page, so roughly 40B link events per day to deduplicate.
  • Raw content storage grows about 30 TB/day.
  • Default politeness: one request per host per 2 seconds. If robots.txt advertises a crawl delay, honor values up to 30 s under this crawler's policy.
  • Freshness target: a URL marked high priority is fetched within 5 minutes of becoming due, and p95 of fetched documents reach the index within 2 minutes.

Out of scope

Ranking and relevance scoring, the user-facing query path, JavaScript rendering (assume a separate headless renderer service), and the legal takedown workflow.