Physical Intelligence

ML Infra Engineer, Compute

Physical Intelligence
Apply
2 days ago

Responsibilities

  • Design and build intelligent multi-tenant scheduling and placement systems based on hardware, topology, availability, cost, and priority.
  • Implement quota management, priority tiers, preemption, queueing, fairness, and resource-sharing policies.
  • Build multi-cluster control-plane capabilities for job lifecycle management, migration, failover, and rescheduling across GPU and TPU clusters.
  • Monitor and optimize accelerator utilization, capacity, cost efficiency, and large-scale training performance.
  • Implement fault detection, automatic recovery, health checking, node management, and scaling for thousands of accelerators.
  • Extend scheduling and orchestration to inference workloads and model deployment on physical robots.
  • Build dashboards, alerting, SLOs, and debugging tools that improve observability and researcher experience.

Requirements

  • Strong software engineering fundamentals.
  • Experience building or operating job scheduling or resource management systems at scale.
  • Experience with large-scale GPU and/or TPU compute clusters.
  • Familiarity with SLURM, Kubernetes, GKE, K3S, or comparable orchestration systems.
  • Ability to reason about resource allocation, bin-packing, priority scheduling, and multi-tenancy.
  • Understanding of long-running, multi-node, straggler-sensitive, and topology-dependent ML training workloads.
  • Ownership of systems end-to-end from design through operation and collaboration with research teams.
  • Bonus: experience with federated scheduling, TPU infrastructure, Borg, YARN, Mesos, Linux systems engineering, networking, infrastructure-as-code, NCCL, collective communication, topology-aware placement, capacity planning, cloud cost optimization, JAX, or PyTorch.
Physical Intelligence

About Physical Intelligence

201-500 employees

Physical Intelligence builds general-purpose AI for the physical world, developing foundation models and full-stack robotic systems for robots and other actuated devices. The privately held company was founded in 2024 and focuses on taking designs from prototype to high-volume production, including actuation, mechatronics, and manufacturing process development. Its work aims to power today’s robots and future hardware across industries.

Contact me