Staff AI Scheduling & Orchestration Engineer

14 days ago

Remote, United States or San Jose, CA, USAStaff+

Responsibilities

Design and implement advanced batch scheduling architectures with Volcano or YuniKorn for multi-node gang scheduling.
Manage cluster-wide admission control and job queueing with Kueue for high-volume AI workloads.
Use Kubernetes Dynamic Resource Allocation and custom scheduler plugins to handle complex accelerator requests.
Architect topology-aware pod placement optimized for NVLink and InfiniBand communication.
Implement GPU sharing technologies including MIG and time-slicing, along with multi-tenancy isolation policies.
Integrate the scheduling layer with bare-metal GPU hardware and storage I/O patterns in collaboration with GPU Systems and Storage teams.
Improve scheduling-stack reliability and scalability by resolving resource contention and deadlocks in large-scale HPC environments.
Mentor junior engineers and conduct design reviews for the orchestration layer.

Bachelor’s or master’s degree in Computer Science, Electrical Engineering, or a related field.
At least 6 years of distributed systems engineering experience.
Deep hands-on expertise with Kubernetes scheduling frameworks and orchestrators.
Experience with AI workload execution patterns and distributed training frameworks such as PyTorch Distributed, Ray, or MPI.
Track record of operating, debugging, and scaling scheduling stacks in HPC or large-scale production cloud environments.
Strong knowledge of GPU hardware architectures and scheduling challenges for distributed AI training and inference.
Experience with infrastructure automation and infrastructure-as-code, including Terraform or Go-based Operators.
Strong technical communication and leadership skills, with the ability to influence cross-functional teams and align architectural goals.
Ability to work in a high-velocity engineering environment and translate ambiguous requirements into scalable engineering solutions.

201-500 employees