
ML Infra Engineer, Compute
Physical Intelligence2 days ago
Responsibilities
- Design and build intelligent multi-tenant scheduling and placement systems based on hardware, topology, availability, cost, and priority.
- Implement quota management, priority tiers, preemption, queueing, fairness, and resource-sharing policies.
- Build multi-cluster control-plane capabilities for job lifecycle management, migration, failover, and rescheduling across GPU and TPU clusters.
- Monitor and optimize accelerator utilization, capacity, cost efficiency, and large-scale training performance.
- Implement fault detection, automatic recovery, health checking, node management, and scaling for thousands of accelerators.
- Extend scheduling and orchestration to inference workloads and model deployment on physical robots.
- Build dashboards, alerting, SLOs, and debugging tools that improve observability and researcher experience.
Requirements
- Strong software engineering fundamentals.
- Experience building or operating job scheduling or resource management systems at scale.
- Experience with large-scale GPU and/or TPU compute clusters.
- Familiarity with SLURM, Kubernetes, GKE, K3S, or comparable orchestration systems.
- Ability to reason about resource allocation, bin-packing, priority scheduling, and multi-tenancy.
- Understanding of long-running, multi-node, straggler-sensitive, and topology-dependent ML training workloads.
- Ownership of systems end-to-end from design through operation and collaboration with research teams.
- Bonus: experience with federated scheduling, TPU infrastructure, Borg, YARN, Mesos, Linux systems engineering, networking, infrastructure-as-code, NCCL, collective communication, topology-aware placement, capacity planning, cloud cost optimization, JAX, or PyTorch.
Tech Stack
Categories
About Physical Intelligence
Physical Intelligence builds general-purpose AI for the physical world, developing foundation models and full-stack robotic systems for robots and other actuated devices. The privately held company was founded in 2024 and focuses on taking designs from prototype to high-volume production, including actuation, mechatronics, and manufacturing process development. Its work aims to power today’s robots and future hardware across industries.