Software Engineer, Data Infra
Thinking Machines Lab5 hours ago
Base Salary
$300k - $400k/yr
Responsibilities
- Design, build, and operate scalable, fault-tolerant infrastructure for distributed LLM research workloads across compute, orchestration, and multimodal storage.
- Develop high-throughput systems for data ingestion, processing, transformation, training-data catalogs, deduplication, quality checks, and search.
- Build traceability, reproducibility, and quality-control systems across the data lifecycle.
- Implement and maintain monitoring and alerting for platform reliability and performance.
- Collaborate with research teams to improve data quality, accelerate training cycles, and deliver new capabilities.
- Own projects end-to-end and contribute across different technology stacks and teams.
Requirements
- Bachelor’s degree or equivalent experience in computer science, engineering, or a similar field.
- Proficiency in at least one backend language, specifically Python or Rust.
- Fluency with distributed compute frameworks such as Apache Spark or Ray.
- Deep familiarity with cloud infrastructure, data lake architectures, and batch and streaming pipelines.
- Preferred hands-on experience with Kafka, dbt, Terraform, and Airflow.
- Preferred experience building a web crawler.
- Preferred extensive experience scaling deduplication, data mining, and search systems.
- Strong knowledge of file formats and storage systems such as Parquet and Delta Lake.
- Ability to work collaboratively with cross-functional partners, take initiative, and own projects end-to-end.
Benefits
- The role is based in San Francisco, California.
- Visa sponsorship is available, with the company committed to working through the visa process for the right fit, though success is not guaranteed for every candidate or role.
- Generous health, dental, and vision benefits.
- Unlimited paid time off.
- Paid parental leave.
- Relocation support as needed.
Tech Stack
Categories
BackendData Engineering
About Thinking Machines Lab
Thinking Machines Lab develops AI and generative AI software and conducts applied research to help organizations make data-driven decisions. The company builds products and data science solutions for enterprise use cases, pairing foundational models with practical tooling and services across industries. It is privately held and headquartered in San Francisco.