3 months ago
Palo Alto, CA, USASenior
Base Salary
$195k - $262k/yr
Responsibilities
- Build and maintain distributed training infrastructure for SFT, continued pretraining, preference optimization, and reinforcement-learning workloads.
- Integrate and extend large-scale training and RL frameworks and internal systems.
- Implement and debug tensor, pipeline, sequence/context, expert, and data parallelism strategies.
- Build rollout, reward-model serving, replay/data buffer, checkpointing, evaluation, and experiment-orchestration components.
- Profile and improve GPU utilization, communication efficiency, memory usage, and training throughput.
- Diagnose failures across NCCL, CUDA, PyTorch, Ray, schedulers, storage, networking, and checkpointing layers.
- Create reproducible training runs, launch scripts, dashboards, runbooks, and operational tooling for research users.
- Partner with research scientists to convert algorithmic training recipes into scalable and debuggable systems.
- Write design documents, incident reports, benchmark reports, and operating guides.
Requirements
- Strong Python and PyTorch engineering skills.
- Hands-on experience with distributed model training, large-scale ML systems, or GPU cluster workloads.
- Practical understanding of transformer training bottlenecks, memory pressure, gradient and optimizer state, communication overhead, and checkpointing.
- Experience debugging production or research training jobs across multiple GPUs or nodes.
- Ability to reason quantitatively about throughput, utilization, memory, reliability, cost, and research velocity.
- Strong communication and collaboration skills with researchers, ML engineers, platform engineers, and leadership.
- Experience with Megatron-LM, DeepSpeed, PyTorch FSDP/DTensor, Ray, Slurm, Kubernetes, or large internal training platforms is preferred.
- Experience with reinforcement-learning infrastructure frameworks such as verl, slime, AReaL, OpenRLHF, TRL, or custom PPO, GRPO, or RLHF systems is preferred.
- Familiarity with NCCL, CUDA, Triton, Nsight, InfiniBand, RDMA, RoCE, H100, H200, B200 clusters, or storage and network bottlenecks is preferred.
- Experience supporting SFT, DPO, PPO, GRPO, RLAIF, reward-model serving, rollout generation, or agent-training workloads is preferred.
- Open-source contributions to distributed training, RL infrastructure, PyTorch, Ray, Megatron, DeepSpeed, or related systems are preferred.
Benefits
- 100% company-paid medical, dental, and vision coverage for employees and families.
- 401(k) plan with up to 4% company match and immediate vesting.
- 20 weeks of paid parental leave for primary caregivers and 12 weeks for secondary caregivers.
- Remote work reimbursement of up to $85 per month for mobile and internet.
- Company-paid short-term disability, long-term disability, and life insurance.
- Remote work is supported, with applicants required to be authorized to work in the country where they apply.
Tech Stack
Categories
About Nebius
Nebius builds a full-stack AI cloud offering GPU compute, storage, and tools for training and deploying ML models for startups, enterprises, and research labs. It sells consumption-based cloud infrastructure (IaaS/PaaS) and managed services tailored to generative AI workloads, including large-scale model training and inference. The company is headquartered in Amsterdam and operates as an independent provider.