Base Salary
$250k - $445k/yr
Responsibilities
- Profile end-to-end training runs to identify compute, communication, and storage bottlenecks.
- Optimize GPU utilization and throughput for large-scale distributed model training.
- Improve kernel efficiency, scheduling, and collective communication performance with runtime and systems engineers.
- Implement model graph transforms to improve end-to-end throughput.
- Build tooling to monitor and visualize MFU, throughput, and uptime across clusters.
- Partner with researchers to ensure new model architectures scale efficiently during pre-training.
- Contribute to infrastructure decisions that improve the reliability and efficiency of large training jobs.
Requirements
- Strong programming skills in Python and C++; Rust or CUDA experience is a plus.
- Experience running distributed training jobs on multi-GPU systems or HPC clusters.
- Exposure to PyTorch, JAX, or TensorFlow and understanding of large-scale training loops.
- Ability to debug complex distributed systems, rigorously measure efficiency, and translate profiling data into engineering improvements.
- Familiarity with NCCL, MPI, or UCX is preferred.
- Experience with large-scale data loading and checkpointing systems is preferred.
- Prior work on training runtime, distributed scheduling, or ML compiler optimization is preferred.
Benefits
- Hybrid work model with three days in the San Francisco office per week
- Relocation assistance for new employees
About OpenAI
OpenAI is an AI research and deployment company dedicated to ensuring that general-purpose artificial intelligence benefits all of humanity. AI is an extremely powerful tool that must be created with safety and human needs at its core. OpenAI is dedicated to putting that alignment of interests first — ahead of profit. To achieve our mission, we must encompass and value the many different perspectives, voices, and experiences that form the full spectrum of humanity. Our investment in diversity, equity, and inclusion is ongoing, executed through a wide range of initiatives, and championed and supported by leadership. At OpenAI, we believe artificial intelligence has the potential to help people solve immense global challenges, and we want the upside of AI to be widely shared. Join us in shaping the future of technology.