DrawLint.aiMedium✦ Official
40 minWeb Crawler
Design a scalable web crawler that systematically browses the internet and builds an index for a search engine.
These requirements and scale numbers are for reference. The AI reviewer will evaluate your design based on the functional requirements and assumptions you define.
🎯 Key Requirements
- •Crawl a target seed list and follow links breadth-first
- •Honor robots.txt and per-domain rate limits
- •Deduplicate URLs and detect duplicate page content
📐 Scale Expectations
- •10M pages crawled per day
- •Re-crawl popular pages every few hours
- •Stay within a fixed bandwidth and politeness budget
🔗 Related Topics
Community Submissions (0)
No designs submitted yet
Be the first!