Software Engineer, Evaluation Platform / Infra
Thinking Machines Lab2 months ago
San Francisco, CA, USA or New York, NY, USAMid Level
H1B sponsor
Base Salary
$300k - $350k/yr
Responsibilities
- Design, build, and maintain a platform for authoring, running, tracking, and analyzing model evaluations.
- Develop evaluation libraries, distributed backend systems, data pipelines, APIs, and user-facing applications end to end.
- Create flexible abstractions for evaluation tasks, environments, graders, datasets, and model outputs.
- Ensure evaluation results are reproducible and trustworthy through versioning, provenance, observability, failure recovery, and quality controls.
- Partner with researchers and cross-functional teams to turn bespoke evaluation workflows into self-service systems.
Requirements
- Bachelor’s degree or equivalent practical experience in computer science, engineering, machine learning, or a related field.
- At least two years of post-graduate software engineer or ML engineer experience, excluding internships.
- Hands-on experience building or maintaining evaluations, benchmarks, graders, or model-quality systems for large language or multimodal models.
- Strong software engineering fundamentals and experience building reliable, maintainable systems.
- Proficiency in at least one backend programming language, with Python and Rust used primarily.
- Experience with databases, data pipelines, distributed systems, or other data-intensive infrastructure.
- Ability to work across the stack and own projects from problem discovery through deployment and operation.
- Experience collaborating with cross-functional partners and subject-matter experts.
- Preferred experience includes building frameworks, SDKs, or developer tools; distributed job execution; workflow orchestration; sandboxed environments; large-scale data processing; polished data-inspection interfaces; model-based or human evaluation; synthetic data; research collaboration; and startup or small-team product development.
Benefits
- Generous health, dental, and vision benefits.
- Unlimited paid time off.
- Paid parental leave.
- Relocation support as needed.
- Role is based in San Francisco, California, or New York, New York.
Tech Stack
Categories
BackendData Engineering
About Thinking Machines Lab
Thinking Machines Lab develops AI and generative AI software and conducts applied research to help organizations make data-driven decisions. The company builds products and data science solutions for enterprise use cases, pairing foundational models with practical tooling and services across industries. It is privately held and headquartered in San Francisco.