Research Engineer, Web Crawling
Thinking Machines Lab1 month ago
Base Salary
$350k - $475k/yr
Responsibilities
- Design and scale web crawlers and ingestion infrastructure for pretraining data.
- Build large-scale extraction, deduplication, and data quality filtering pipelines.
- Develop specialized crawlers for high-value or difficult-to-reach data sources.
- Collaborate with the pretraining team to assess how crawled data affects model performance.
- Improve the reliability and efficiency of crawling and ingestion infrastructure at petabyte scale.
- Help set technical direction and share crawling expertise with other engineers.
Requirements
- 8+ years of experience designing, building, and scaling web crawlers, scrapers, or large-scale distributed data-acquisition systems.
- Demonstrated ownership of crawler or data-acquisition infrastructure at internet scale.
- Strong software engineering skills in Python, Go, or Rust, with practical distributed-systems experience.
- Working knowledge of practical and legal considerations for large-scale web data collection, including robots.txt, rate limiting, and licensing.
- Preferred: experience applying machine learning to crawl selection, extraction, or data quality classification at internet scale.
- Preferred: experience setting technical direction for crawling, data acquisition, or search infrastructure teams.
- Preferred: experience designing petabyte-scale storage and processing systems.
- Preferred: open-source contributions to crawling, scraping, or data infrastructure tools.
- Preferred: background at a search engine or frontier AI lab data-acquisition team.
Benefits
- Annual salary range of $350,000-$475,000 USD.
- Visa sponsorship is available, with commitment to work through the visa process for qualified candidates, though success is not guaranteed for every candidate or role.
- Generous health, dental, and vision benefits.
- Unlimited paid time off and paid parental leave.
- Relocation support as needed.
- Based in San Francisco, CA or New York, NY.
About Thinking Machines Lab
Thinking Machines Lab develops AI and generative AI software and conducts applied research to help organizations make data-driven decisions. The company builds products and data science solutions for enterprise use cases, pairing foundational models with practical tooling and services across industries. It is privately held and headquartered in San Francisco.