Software Engineer, Evaluation Platform / Infra
Thinking Machines Lab1 month ago
Base Salary
$300k - $475k/yr
Responsibilities
- Design, build, and maintain a platform for authoring, running, tracking, and analyzing model evaluations.
- Develop evaluation libraries, distributed backend systems, data pipelines, APIs, and user-facing applications end to end.
- Create flexible abstractions for evaluation tasks, environments, graders, datasets, and model outputs.
- Ensure evaluation results are reproducible and trustworthy through versioning, provenance, observability, failure recovery, and quality controls.
- Partner with researchers to identify bottlenecks and convert bespoke workflows into self-serve systems.
- Collaborate with the Research Tooling team and other engineering and research groups.
Requirements
- Bachelor’s degree or equivalent practical experience in computer science, engineering, machine learning, or a related field.
- At least two years of post-graduation experience as a software engineer or ML engineer, excluding internships.
- Hands-on experience building or maintaining evaluations, benchmarks, graders, or model-quality systems for large language or multimodal models.
- Strong software engineering fundamentals and experience building reliable, maintainable systems.
- Proficiency in at least one backend programming language, with Python and Rust used primarily.
- Experience with databases, data pipelines, distributed systems, or other data-intensive infrastructure.
- Ability to work across the stack and own projects from problem discovery through deployment and operation.
- Experience collaborating with cross-functional partners and subject-matter experts.
- Preferred experience includes building frameworks, SDKs, or developer tools; distributed job execution; workflow orchestration; sandboxed environments; large-scale data processing; polished interfaces for complex data; model evaluation; research collaboration; or technically complex startup products.
Benefits
- Generous health, dental, and vision benefits.
- Unlimited paid time off.
- Paid parental leave.
- Relocation support as needed.
- Visa sponsorship is available.
- Based in San Francisco, California.
Tech Stack
Categories
BackendData Engineering
About Thinking Machines Lab
Thinking Machines Lab develops AI and generative AI software and conducts applied research to help organizations make data-driven decisions. The company builds products and data science solutions for enterprise use cases, pairing foundational models with practical tooling and services across industries. It is privately held and headquartered in San Francisco.