AI Training Infrastructure Engineer
Designworks Talent14 days ago
Bellevue, WA, USASenior / Staff+
Responsibilities
- Build and scale distributed training infrastructure for large AI models across GPU clusters.
- Design systems that improve training reliability, efficiency, fault tolerance, checkpointing, recovery, and resource utilization.
- Integrate AI models into production training pipelines with platform, orchestration, performance, and machine learning teams.
- Diagnose issues affecting training throughput, stability, reliability, and cost efficiency.
- Build automation and developer tooling for AI researchers and engineers.
- Establish training infrastructure, operational, and platform reliability best practices.
- Contribute to the architecture and evolution of the AI infrastructure platform.
Requirements
- Hands-on experience building and operating distributed training systems or large-scale machine learning infrastructure.
- Experience supporting large AI models, foundation models, post-training workflows, or similar machine learning systems.
- Strong understanding of reliability, scalability, and efficiency challenges in multi-node GPU training.
- Experience integrating training systems with production machine learning pipelines.
- Strong programming skills and experience with complex distributed systems.
- Ability to independently own technically challenging projects in a fast-moving, high-ownership environment.
- Preferred experience with PyTorch Distributed, DeepSpeed, Megatron-LM, Ray, or similar distributed training technologies.
- Preferred experience with supervised fine-tuning, reinforcement learning from human feedback, or other post-training workflows.
- Preferred background operating AI training infrastructure at scale in a hyperscaler, AI research organization, cloud provider, or GPU cloud environment.
- Preferred experience optimizing GPU utilization, training performance, or distributed-system reliability.
- Familiarity with Kubernetes, containerized AI workloads, and large-scale infrastructure platforms.
Benefits
- Competitive base pay for the Bellevue market, with certain roles eligible for merit increases, annual bonuses, and long-term incentives.
- Medical, dental, and vision insurance for U.S.-based employees.
- 401(k) plan with company match.
- Paid holidays per calendar year.
- Hybrid role in downtown Bellevue, Washington, with approximately three days per week in the office.
- Candidates elsewhere in the U.S. may apply if open to relocation.
Tech Stack
Categories
About Designworks Talent
Designworks Talent specializes in the design and delivery of enterprise workforce solutions including strategy, talent acquisition, engagement, and succession planning. We work collaboratively with business leaders and enterprise Talent Acquisition and Human Resources teams to deliver custom workforce solutions to meet the dynamic talent demands of your business.