DrawLintDrawLint.ai
Medium✦ Official
40 min

Web Crawler

Design a scalable web crawler that systematically browses the internet and builds an index for a search engine.

These requirements and scale numbers are for reference. The AI reviewer will evaluate your design based on the functional requirements and assumptions you define.

🎯 Key Requirements

  • Crawl a target seed list and follow links breadth-first
  • Honor robots.txt and per-domain rate limits
  • Deduplicate URLs and detect duplicate page content

📐 Scale Expectations

  • 10M pages crawled per day
  • Re-crawl popular pages every few hours
  • Stay within a fixed bandwidth and politeness budget

🔗 Related Topics

Community Submissions (0)

No designs submitted yet

Be the first!

Go to Canvas