1 year ago
Base Salary
$150k - $300k/yr
Responsibilities
- Build a distributed crawler capable of handling more than 100 million pages per day.
- Optimize crawl politeness and rate limiting across thousands of domains.
- Design systems to detect and handle dynamic content, JavaScript rendering, and anti-bot measures.
- Create intelligent crawl scheduling and prioritization algorithms for maximum coverage efficiency.
Requirements
- Extensive experience building and scaling web crawlers, or the ability to ramp up very quickly.
- Experience with a high-performance language such as C++ or Rust.
- Familiarity with TypeScript, Playwright, modern web design, and CDP (Chrome DevTools Protocol).
- Ability to optimize systems to an exceptional degree.
- Interest in finding high-quality knowledge and its importance to the world.
Benefits
- In-person work opportunity in San Francisco.
- International candidate sponsorship is available, including STEM OPT, OPT, H1B, O1, and E3.
- Premium medical, dental, and vision healthcare benefits.
- Fertility benefits.
- Monthly wellness stipend.
Tech Stack
Categories
About Exa
Exa builds a web-scale search engine and API for AI agents and developer applications, combining its own crawler, embedding models, and high-performance vector search. It sells usage-based APIs and enterprise integrations that power retrieval-augmented generation, browsing, and automation on live web content. The company runs large crawling infrastructure and dedicated GPU clusters to continuously index and embed pages, and ships low-latency search components written in Rust.