1 year ago
Base Salary
$150k - $300k/yr
Responsibilities
- Build a distributed crawler capable of handling more than 100 million pages per day.
- Optimize crawl politeness and rate limiting across thousands of domains.
- Design systems to detect and handle dynamic content, JavaScript rendering, and anti-bot measures.
- Create intelligent crawl scheduling and prioritization algorithms for maximum coverage efficiency.
Requirements
- Extensive experience building and scaling web crawlers, or the ability to ramp up very quickly.
- Experience with a high-performance language such as C++ or Rust.
- Familiarity with TypeScript, Playwright, modern web design, and CDP (Chrome DevTools Protocol).
- Ability to optimize systems to an exceptional degree.
- Interest in finding high-quality knowledge and its importance to the world.
Benefits
- In-person work opportunity in San Francisco.
- International candidate sponsorship is available, including STEM OPT, OPT, H1B, O1, and E3.
- Premium medical, dental, and vision healthcare benefits.
- Fertility benefits.
- Monthly wellness stipend.
Tech Stack
Categories
About Exa
Exa is an applied AI research lab organizing human knowledge, starting with the web. Exa Search is the highest quality search API at every latency. It accepts natural language queries and produces token-efficient results with citations. Exa Agent is a high-compute web agent that synthesizes information from across the web to handle the most complex deep research, list-building, and enrichment workflows. These products power Cursor, Cognition, Hubspot, OpenRouter, Monday.com, and agents built by over 400,000 developers. We are a worldwide team of engineers and researchers. We're hiring: exa.ai/careers