Staff AI Training Infrastructure Engineer
Designworks Talent4 days ago
Bellevue, WA, USAStaff+
Responsibilities
- Build and scale distributed training infrastructure for large AI models across large GPU clusters.
- Design systems that improve training reliability, efficiency, and resource utilization.
- Develop fault-tolerance, checkpointing, recovery, and large-scale training-operation solutions.
- Integrate AI models into production training pipelines with platform, orchestration, and performance engineering teams.
- Diagnose and resolve issues affecting training throughput, stability, reliability, and cost efficiency.
- Build tools and automation that improve the developer experience for AI researchers and engineers.
- Establish best practices for training infrastructure, operational processes, and platform reliability.
- Contribute to the AI infrastructure platform as an early engineering team member.
Requirements
- Hands-on experience building and operating distributed training systems or large-scale machine learning infrastructure.
- Experience supporting large AI models, foundation models, post-training workflows, or similar machine learning systems.
- Strong understanding of reliability, scalability, and efficiency challenges in multi-node GPU training.
- Experience integrating training systems with production machine learning pipelines.
- Strong programming skills and experience with complex distributed systems.
- Ability to independently own technically challenging projects in a fast-moving engineering environment.
- Experience with distributed training frameworks such as PyTorch Distributed, DeepSpeed, Megatron-LM, Ray, or similar technologies is preferred.
- Experience with supervised fine-tuning, reinforcement learning from human feedback, or other post-training workflows is preferred.
- Background operating AI training infrastructure at scale within a hyperscaler, AI research organization, cloud provider, or GPU cloud environment is preferred.
- Experience optimizing GPU utilization, training performance, or distributed-system reliability is preferred.
- Familiarity with Kubernetes, containerized AI workloads, and large-scale infrastructure platforms is preferred.
Benefits
- Competitive base pay for the Bellevue market.
- Medical, dental, and vision insurance for U.S.-based employees.
- 401(k) plan and company match.
- Paid holidays per calendar year.
- Hybrid role based in the Bellevue, WA area with approximately three days per week in the office.
- Candidates elsewhere in the U.S. may apply if open to relocation.
Tech Stack
Categories
About Designworks Talent
Designworks Talent specializes in the design and delivery of enterprise workforce solutions including strategy, talent acquisition, engagement, and succession planning. We work collaboratively with business leaders and enterprise Talent Acquisition and Human Resources teams to deliver custom workforce solutions to meet the dynamic talent demands of your business.