Research Engineer - Distributed Training
Prime Intellect4 months ago
Base Salary
$150k - $350k/yr
Responsibilities
- Build and optimize distributed training infrastructure for pre-training and large-scale RL workloads through the prime-rl framework.
- Improve end-to-end training efficiency across compute, memory, networking, and scheduling layers.
- Design and implement low-level optimizations for kernels, communication paths, and runtimes.
- Develop distributed training systems spanning data, tensor, and pipeline parallel workloads.
- Shape RL training infrastructure, including asynchronous rollout and post-training systems.
- Contribute to open-source libraries and internal infrastructure for frontier-scale model training.
- Collaborate with researchers and infrastructure engineers to turn bottlenecks into systems improvements.
- Work at the frontier of training systems, inference systems, compiler/runtime tooling, and hardware-aware optimization.
Requirements
- Strong systems engineering experience in AI/ML infrastructure, especially large-scale model training or inference.
- Deep familiarity with PyTorch and distributed training frameworks such as PyTorch Distributed, DeepSpeed, FSDP, Megatron, vLLM, Ray, or related tools.
- Experience optimizing performance across kernels, memory movement, communication overhead, or parallelization strategy.
- Hands-on experience with data, tensor, and pipeline parallelism.
- Strong understanding of GPU architecture, profiling, and performance debugging.
- Ability to identify cross-stack bottlenecks and drive improvements from first principles.
- Optional qualifications include CUDA or Triton kernel optimization, ML compiler/runtime optimization, RL training infrastructure, multi-node GPU clusters, high-performance networking, open-source ML systems contributions, and technical publishing.
Benefits
- Cash compensation range of $150-350k, including equity incentives.
- Flexible work arrangements with remote or in-person work available at the San Francisco offices.
- Visa sponsorship and relocation assistance for international candidates.
- Quarterly team off-sites, hackathons, conferences, and learning opportunities.
Tech Stack
Categories
About Prime Intellect
Prime Intellect builds a full-stack platform for training frontier AI models, giving enterprises access to distributed compute and agentic training infrastructure to develop their own systems. The company operates as a platform and open research lab, with customers leveraging its infrastructure rather than building it in-house. Headquartered in San Francisco and privately held, it has raised a Series A round backed by investors including Founders Fund and NVIDIA.