Site Reliability Engineer, Post Training
Thinking Machines Lab2 months ago
San Francisco, CA, USA or New York, NY, USAMid Level
H1B sponsor
Base Salary
$300k - $350k/yr
Responsibilities
- Own the reliability, performance, and uptime of large-scale post-training and reinforcement learning jobs from launch through completion.
- Partner with research teams during active model runs to debug failures, unblock training, and accelerate iteration.
- Debug issues across accelerators, networking, storage, schedulers, and training frameworks and drive them to root cause.
- Build monitoring, alerting, automated recovery, checkpointing, and fault-tolerance systems for training workloads.
- Improve job scheduling, cluster utilization, and internal tooling while reducing operational toil.
- Participate in an on-call rotation for production model runs and write postmortems that lead to permanent infrastructure fixes.
Requirements
- At least 4 years of experience as a production engineer, site reliability engineer, or infrastructure engineer operating large-scale distributed systems in production.
- Demonstrated ability to debug complex distributed-systems failures involving networking, hardware, kernels, or schedulers.
- Strong software engineering skills in Python and/or Go/C++ with sound judgment about scripting fixes versus building systems.
- Strong grounding in Linux systems internals and networking fundamentals.
- Experience owning production systems and participating in on-call rotations.
- Preferred experience operating GPU or TPU training clusters at scale.
- Preferred familiarity with post-training and reinforcement learning techniques such as RLHF, PPO, and DPO, including reward model serving, rollout generation, and mixed training/inference workloads.
- Preferred experience with PyTorch, Ray, Slurm, Kubernetes, InfiniBand, RDMA, NCCL, and observability tooling for ML training.
Benefits
- San Francisco, California-based role.
- Annual base salary range of $300,000-$350,000 USD.
- Visa sponsorship is available, subject to the company’s stated commitment and visa-process limitations.
- Generous health, dental, and vision benefits, unlimited PTO, paid parental leave, and relocation support as needed.
About Thinking Machines Lab
Thinking Machines Lab develops AI and generative AI software and conducts applied research to help organizations make data-driven decisions. The company builds products and data science solutions for enterprise use cases, pairing foundational models with practical tooling and services across industries. It is privately held and headquartered in San Francisco.