Site Reliability Engineer, Post Training
Thinking Machines Lab1 month ago
San Francisco, CA, USA or New York, NY, USAMid Level
H1B sponsor
Base Salary
$350k - $475k/yr
Responsibilities
- Own the reliability, performance, and uptime of large-scale post-training and reinforcement learning jobs from launch through completion.
- Partner directly with research teams during active model runs to debug issues and accelerate iteration.
- Debug failures across accelerators, networking, storage, schedulers, and training frameworks and drive them to root cause.
- Build monitoring, alerting, automated recovery, checkpointing, fault tolerance, and scheduling improvements.
- Build internal tools that reduce operational toil and improve cluster utilization.
- Participate in an on-call rotation for production model runs and write postmortems that lead to permanent infrastructure fixes.
Requirements
- At least 4 years of experience as a production engineer, site reliability engineer, or infrastructure engineer operating large-scale distributed systems in production.
- Demonstrated ability to debug complex distributed-systems failures involving networking, hardware, kernels, or schedulers.
- Strong software engineering skills in Python and/or Go or C++.
- Solid understanding of Linux systems internals and networking fundamentals.
- Experience owning production systems and participating in on-call rotations.
- Preferred: experience operating GPU or TPU training clusters at scale.
- Preferred: familiarity with post-training and reinforcement learning techniques including RLHF, PPO, and DPO.
- Preferred: experience with PyTorch, Ray, Slurm, or Kubernetes and with high-performance networking such as InfiniBand, RDMA, or NCCL.
- Preferred: experience building observability tooling specifically for machine learning training workloads.
Benefits
- Generous health, dental, and vision benefits.
- Unlimited paid time off and paid parental leave.
- Relocation support as needed.
- Visa sponsorship is available.
- The role is based in San Francisco, California.
About Thinking Machines Lab
Thinking Machines Lab develops AI and generative AI software and conducts applied research to help organizations make data-driven decisions. The company builds products and data science solutions for enterprise use cases, pairing foundational models with practical tooling and services across industries. It is privately held and headquartered in San Francisco.