Member of Technical Staff (AI Infrastructure Engineer)
Perplexity7 months ago
London, United KingdomMid Level / Senior
Responsibilities
- Design, deploy, and maintain scalable Kubernetes clusters for AI training and inference workloads.
- Manage and optimize Slurm-based HPC environments, including scheduling, resource allocation, and cluster configuration.
- Develop APIs, orchestration systems, resource schedulers, and job-management systems for batch and real-time workloads.
- Benchmark performance, diagnose bottlenecks, and optimize GPU, compute, storage, networking, and cluster utilization.
- Build monitoring, alerting, and observability solutions for ML workloads running on Kubernetes and Slurm.
- Implement autoscaling strategies and support high-availability inference services and long-running training jobs.
- Respond to outages and collaborate across teams to maintain high uptime for critical training and inference services.
Requirements
- Strong expertise administering Kubernetes, including cluster management, operators, and custom resource definitions.
- Hands-on experience with Slurm workload management, cluster administration, resource allocation, and advanced scheduling.
- Experience deploying and managing distributed training systems and GPU clusters at scale.
- Deep understanding of container orchestration, distributed systems, networking, storage, and compute resource management.
- High-level familiarity with LLM architecture, including Multi-Head Attention, Multi/Grouped-Query, and distributed training strategies.
- Proficiency in Python and C++ for systems and infrastructure automation.
- Hands-on experience with PyTorch in distributed training contexts.
- Strong debugging, monitoring, and observability skills for containerized environments.
- Preferred experience with Kubernetes operators and custom controllers, Slurm federation, CUDA optimization, TensorFlow, distributed training libraries, HPC, parallel computing, high-performance networking, Terraform, Ansible, container registries, image optimization, and multi-stage builds.
- Ideally, 3–5 years of relevant experience in ML systems deployment focused on cluster orchestration and resource management.
- Previous SRE, DevOps, or platform engineering experience focused on ML infrastructure.
Categories
About Perplexity
Perplexity builds an AI-powered answer engine for web and mobile and an enterprise product, Perplexity Computer, for AI-driven workflows across tools and apps. The company monetizes through individual subscriptions (Perplexity Pro) and enterprise plans. Founded in 2022 and headquartered in San Francisco, it operates as a privately held company.