Senior AI Training Infrastructure Engineer
Designworks Talent20 hours ago
Bellevue, WA, USASenior / Staff+
Responsibilities
- Build and scale distributed training infrastructure for large AI models across GPU clusters.
- Design systems that improve training reliability, efficiency, and resource utilization.
- Develop fault-tolerance, checkpointing, recovery, and large-scale training operations solutions.
- Integrate AI models into production training pipelines with platform, orchestration, and performance engineering teams.
- Diagnose issues affecting training throughput, stability, reliability, and cost efficiency.
- Build automation and tools that improve the developer experience for AI researchers and engineers.
- Establish training infrastructure, operational, and platform reliability best practices.
- Contribute to the AI infrastructure platform as an early engineering team member.
Requirements
- Hands-on experience building and operating distributed training systems or large-scale machine learning infrastructure.
- Experience supporting large AI models, foundation models, post-training workflows, or similar ML systems.
- Strong understanding of reliability, scalability, and efficiency challenges in multi-node GPU training.
- Experience integrating training systems with production machine learning pipelines.
- Strong programming skills and experience with complex distributed systems.
- Ability to independently own technically challenging projects in a fast-moving, high-ownership environment.
- Preferred: experience with PyTorch Distributed, DeepSpeed, Megatron-LM, Ray, or similar distributed training technologies.
- Preferred: experience with supervised fine-tuning, reinforcement learning from human feedback, or other post-training workflows.
- Preferred: experience operating AI training infrastructure at scale in a hyperscaler, AI research organization, cloud provider, or GPU cloud environment.
- Preferred: experience optimizing GPU utilization, training performance, or distributed-system reliability.
- Preferred: familiarity with Kubernetes, containerized AI workloads, and large-scale infrastructure platforms.
Benefits
- Medical, dental, and vision insurance for U.S.-based employees.
- 401(k) plan with company match.
- Paid holidays per calendar year.
- Certain roles may be eligible for merit increases, annual bonuses, and long-term incentives.
- Hybrid Bellevue, Washington-area role with approximately three days per week in the office.
- Candidates elsewhere in the U.S. who are open to relocation are encouraged to apply.
Tech Stack
Categories
About Designworks Talent
Designworks Talent specializes in the design and delivery of enterprise workforce solutions including strategy, talent acquisition, engagement, and succession planning. We work collaboratively with business leaders and enterprise Talent Acquisition and Human Resources teams to deliver custom workforce solutions to meet the dynamic talent demands of your business.