Software Engineer, Tinker Platform
Thinking Machines Lab7 hours ago
Base Salary
$350k - $475k/yr
Responsibilities
- Design, build, and operate distributed systems for scheduling, resource orchestration, checkpointing, and fault recovery across large multi-node GPU clusters.
- Own the reliability, latency, and throughput of live production systems serving concurrent training and sampling workloads.
- Implement graceful degradation and fast recovery from node failures, preemptions, and network partitions.
- Improve platform observability through metrics, tracing, and alerting.
- Lead incident response, root-cause analysis, and durable remediation for owned systems.
- Partner with research and ML infrastructure teams to evolve the platform for new training and sampling workloads.
- Make build-versus-buy and architectural decisions for core infrastructure and mentor engineers on distributed-systems practices.
Requirements
- 7+ years of experience building, running, and scaling distributed systems in production with direct on-call ownership of live services.
- Deep understanding of consensus, consistency models, failure detection, scheduling, and state management under partial failure.
- Demonstrated ability to design reliable systems at scale and diagnose and resolve production incidents under time pressure.
- Strong systems-level coding ability in Python, Go, Rust, or C++ and comfort working close to the infrastructure layer.
- Preferred: experience operating large-scale GPU or accelerator clusters, including orchestration, resource scheduling, or ML training infrastructure.
- Preferred: familiarity with large-language-model training or inference stacks and fine-tuning or post-training systems demands.
- Preferred: experience with checkpointing, distributed storage, or high-throughput networking such as RDMA and NCCL.
- Preferred: experience building reliable and usable developer-facing infrastructure or APIs.
- Preferred: experience driving platform-wide architectural decisions and operating autonomously in a fast-changing environment.
Benefits
- Based in San Francisco, California.
- Visa sponsorship is available.
- Health, dental, and vision benefits are offered.
- Unlimited paid time off and paid parental leave are offered.
- Relocation support is available as needed.