Research Engineer, Infrastructure, Training Systems
Thinking Machines Lab3 months ago
Base Salary
$400k - $500k/yr
Responsibilities
- Design, implement, and operate distributed training infrastructure across thousands of GPUs and nodes.
- Improve training throughput, accelerator utilization, and time to train by diagnosing compute, memory, networking, and storage bottlenecks.
- Develop parallelism strategies, runtime components, communication primitives, and workload orchestration.
- Improve reliability through observability, checkpointing, failure diagnosis, and recovery systems.
- Collaborate with researchers to test changes and measure training performance and correctness.
- Build reusable frameworks and libraries that accelerate experimentation and improve reproducibility.
- Share technical findings through documentation, open-source contributions, or technical reports.
Requirements
- Strong engineering skills with the ability to write performant, maintainable code and debug complex systems.
- Hands-on experience building, optimizing, or operating distributed model-training infrastructure.
- Depth in at least one area such as distributed training frameworks, parallelism, GPU communication, low-precision training, runtime systems, or training-workload orchestration.
- Understanding of deep learning frameworks such as PyTorch or JAX and the systems beneath them.
- Ability to investigate issues across subsystem boundaries and carry solutions through implementation and validation.
- Collaborative communication skills, including explaining evidence, uncertainty, and tradeoffs.
- Bachelor’s degree in a relevant technical field or equivalent practical experience.
- Experience with FSDP, ZeRO, model parallelism, mixed precision, FP4/FP8, kernels, compiler work, NCCL, distributed checkpointing, fault tolerance, scheduling, or large-scale training infrastructure is especially relevant.
- Substantial contributions to projects such as PyTorch, XLA, Megatron-LM, DeepSpeed, or related infrastructure are relevant.
Benefits
- Annual base salary range of $400,000 - $500,000 USD.
- Generous health, dental, and vision benefits.
- Unlimited PTO and paid parental leave.
- Relocation support as needed.
- Visa sponsorship is offered, with success not guaranteed for every candidate or role.
- Located in San Francisco, California.
- Evergreen role with continuously reviewed applications.
Categories
About Thinking Machines Lab
Thinking Machines Lab develops AI and generative AI software and conducts applied research to help organizations make data-driven decisions. The company builds products and data science solutions for enterprise use cases, pairing foundational models with practical tooling and services across industries. It is privately held and headquartered in San Francisco.