12 months ago
Base Salary
$150k - $300k/yr
Responsibilities
- Build a distributed crawler capable of handling 100M+ pages per day.
- Optimize crawl politeness and rate limiting across thousands of domains.
- Design systems to detect and manage dynamic content and anti-bot measures.
- Create intelligent crawl scheduling and prioritization algorithms for maximum efficiency.
Requirements
- Extensive experience building and scaling web crawlers or a strong willingness to learn quickly.
- Experience with high-performance programming languages such as C++ or Rust.
- Familiarity with TypeScript, Playwright, and modern web design.
- Comfortable optimizing systems to a high degree.
- Passionate about finding high-quality knowledge and its global importance.
Benefits
- Premium healthcare benefits including medical, dental, and vision.
- Fertility benefits offered to all employees.
- Monthly wellness stipend provided.