Thinking Machines Lab

Software Engineer, Tinker Infrastructure

Thinking Machines Lab
Apply
2 days ago

Base Salary

$300k - $400k/yr

Responsibilities

  • Design, build, and operate distributed systems for scheduling, resource orchestration, checkpointing, and fault recovery across large GPU clusters.
  • Own the reliability, latency, and throughput of production systems serving concurrent training and sampling workloads.
  • Design graceful degradation and rapid recovery for node failures, preemptions, network partitions, and other partial failures.
  • Improve platform observability through metrics, tracing, and alerting.
  • Lead incident response, root-cause analysis, and durable remediation for owned systems.
  • Partner with research and ML infrastructure teams to evolve the platform for new training and sampling workloads.
  • Make architectural and build-versus-buy decisions for core infrastructure and mentor engineers on distributed-systems practices.

Requirements

  • 7+ years of experience building, operating, and scaling distributed systems in production with direct on-call ownership of live services.
  • Deep understanding of consensus, consistency models, failure detection, scheduling, and state management under partial failure.
  • Demonstrated ability to design reliable systems at scale and diagnose and resolve production incidents under time pressure.
  • Strong systems-level coding ability in Python, Go, Rust, or C++, with comfort working close to the infrastructure layer.
  • Experience with large-scale GPU or accelerator clusters, job orchestration, resource scheduling, or ML training infrastructure is preferred.
  • Familiarity with large-language-model training or inference systems and fine-tuning or post-training workload demands is preferred.
  • Experience with checkpointing, distributed storage, or high-throughput networking such as RDMA or NCCL is preferred.
  • Experience building reliable and usable developer-facing infrastructure or APIs is preferred.
  • Platform-wide architectural leadership and significant autonomy in a fast-changing environment are preferred.

Benefits

  • Annual salary range of $300,000-$400,000 USD.
  • Health, dental, and vision benefits.
  • Unlimited paid time off and paid parental leave.
  • Relocation support as needed.
  • Role is based in San Francisco, California, or New York, New York.
  • Visa sponsorship is available with a commitment to work through the visa process together, although success is not guaranteed for every candidate or role.

Tech Stack

Categories

Thinking Machines Lab

About Thinking Machines Lab

201-500 employees

Thinking Machines Lab develops AI and generative AI software and conducts applied research to help organizations make data-driven decisions. The company builds products and data science solutions for enterprise use cases, pairing foundational models with practical tooling and services across industries. It is privately held and headquartered in San Francisco.

Contact me