Thinking Machines Lab

Software Engineer, Training Infrastructure

Thinking Machines Lab
Apply
2 days ago
San Francisco, CA, USA or New York, NY, USAMid Level
H1B sponsor

Base Salary

$300k - $400k/yr

Responsibilities

  • Design, build, and operate distributed training infrastructure for Tinker and internal frontier-model training, including scheduling, orchestration, checkpointing, and recovery.
  • Improve training throughput and cost efficiency through profiling, parallelism strategies, memory optimization, and kernel- and communication-level tuning.
  • Support new open-weights and in-house model architectures, including mixture-of-experts models.
  • Build automated failure detection, job recovery, multi-tenant isolation, and observability for long-running training jobs.
  • Support multi-tenant LoRA fine-tuning and tightly coupled sampling and training for reinforcement learning workloads.
  • Collaborate with Tinker API, frameworks, and product teams to maintain consistency between infrastructure and the user-facing SDK.

Requirements

  • 4+ years of software engineering experience, including building and operating production distributed systems.
  • Strong Python programming skills and experience with a systems language such as C++, Rust, or Go.
  • Hands-on experience with large-scale model training or inference infrastructure using frameworks such as PyTorch.
  • Experience debugging complex performance and reliability issues in distributed environments.
  • Preferred: experience training or serving large language models at scale, including mixture-of-experts architectures.
  • Preferred: experience with LoRA, parameter-efficient fine-tuning, or supervised and reinforcement learning pipelines for LLMs.
  • Preferred: experience with cluster orchestration and scheduling, such as Kubernetes or Slurm, and fault-tolerant checkpointing.
  • Preferred: experience building observability and profiling tooling.
  • Preferred: contributions to open-source ML systems projects such as PyTorch, Megatron, vLLM, or SGLang.

Benefits

  • Based in San Francisco, California or New York, New York.
  • Visa sponsorship is offered, subject to the stated qualification and visa-process caveat.
  • Generous health, dental, and vision benefits.
  • Unlimited PTO and paid parental leave.
  • Relocation support as needed.
Thinking Machines Lab

About Thinking Machines Lab

201-500 employees

Thinking Machines Lab develops AI and generative AI software and conducts applied research to help organizations make data-driven decisions. The company builds products and data science solutions for enterprise use cases, pairing foundational models with practical tooling and services across industries. It is privately held and headquartered in San Francisco.

Contact me