Web Crawling - Research Engineer
Thinking Machines Lab5 days ago
Base Salary
$350k - $475k/yr
Responsibilities
- Design and scale web crawlers and ingestion infrastructure for pretraining data.
- Build large-scale extraction, deduplication, and data-quality filtering pipelines.
- Develop specialized crawlers for high-value or difficult-to-reach data sources.
- Collaborate with pretraining teams to understand how crawled data affects model performance.
- Improve the reliability and efficiency of crawling and ingestion infrastructure at petabyte scale.
- Set technical direction for the crawling area and help other engineers build expertise.
Requirements
- 8+ years of experience designing, building, and scaling web crawlers, scrapers, or large-scale distributed data-acquisition systems.
- Demonstrated ownership of crawler or data-acquisition infrastructure at internet scale.
- Strong software engineering skills in Python, Go, or Rust, with practical distributed-systems experience.
- Working knowledge of practical and legal considerations for large-scale web data collection, including robots.txt, rate limiting, and licensing.
- Preferred: experience applying machine learning to crawl selection, extraction, or data-quality classification at internet scale.
- Preferred: experience setting technical direction for crawling, data acquisition, or search infrastructure teams.
- Preferred: experience designing petabyte-scale storage and processing systems.
- Preferred: open-source contributions to crawling, scraping, or data-infrastructure tools.
- Preferred: background in search-engine crawling or indexing, or in a frontier AI lab's data-acquisition team.
Benefits
- Based in San Francisco, California.
- Annual base salary range of $350,000-$475,000 USD.
- Visa sponsorship is available, with a commitment to work through the visa process for qualified candidates, though success is not guaranteed for every candidate or role.
- Generous health, dental, and vision benefits.
- Unlimited paid time off.
- Paid parental leave.
- Relocation support as needed.
About Thinking Machines Lab
Thinking Machines Lab develops AI and generative AI software and conducts applied research to help organizations make data-driven decisions. The company builds products and data science solutions for enterprise use cases, pairing foundational models with practical tooling and services across industries. It is privately held and headquartered in San Francisco.