Site Reliability Engineer, Production
Thinking Machines Lab2 months ago
Base Salary
$300k - $350k/yr
Responsibilities
- Own end-to-end reliability for Tinker, from CI/CD flows through production observability and incident response.
- Define Service Level Objectives for distributed training systems, balancing job completion reliability, scheduling latency, and development velocity.
- Design and implement monitoring and observability across the full training path.
- Lead incident response, recovery, postmortems, and systematic reliability improvements for Tinker platform issues.
- Harden multi-tenant isolation and resource scheduling for LoRA-based workload co-scheduling without compromising reliability or data separation.
- Collaborate with security teams to address production vulnerabilities.
Requirements
- Bachelor’s degree or equivalent experience in computer science, engineering, or a similar field.
- Experience with distributed systems, cloud infrastructure, or site reliability engineering.
- Proficiency writing software for reliability problems, including tooling and automation.
- Experience with production incident response, postmortems, and systematic reliability improvement.
- Strong communication skills and a track record of coordination across engineering and research teams.
- Preferred: deep experience operating production cloud services at scale.
- Preferred: background in distributed training frameworks and infrastructure failure behavior in training workloads.
- Preferred: experience building checkpoint and recovery systems for long-running distributed jobs.
- Preferred: expertise deploying, operating, debugging, and tuning Kubernetes clusters for heterogeneous GPU workloads.
Benefits
- Based in San Francisco, California.
- Visa sponsorship is offered, subject to the stated visa-process qualification caveat.
- Generous health, dental, and vision benefits.
- Unlimited paid time off and paid parental leave.
- Relocation support as needed.
Tech Stack
Categories
Site Reliability
About Thinking Machines Lab
Thinking Machines Lab develops AI and generative AI software and conducts applied research to help organizations make data-driven decisions. The company builds products and data science solutions for enterprise use cases, pairing foundational models with practical tooling and services across industries. It is privately held and headquartered in San Francisco.