2 months ago
Base Salary
$227k - $356k/yr
Responsibilities
- Build and operate monitoring and alerting for fabric, GPU, power, thermal, and job-level cluster health signals.
- Remotely deploy and configure large-scale HPC clusters for AI workloads using automation.
- Automate operating system, firmware, driver, and networking lifecycle management with Ansible and Terraform.
- Create runbooks and automated remediations for common cluster failure modes.
- Troubleshoot InfiniBand, RoCE, CLOS fabrics, 100GbE, Ethernet switching, GPU-direct, NCCL, fabric, power, and cluster issues.
- Participate in on-call rotations and lead incident response for cluster-level problems.
- Maintain standard operating procedures and provide requirements to engineering teams focused on stability and operational efficiency.
Requirements
- 7+ years of experience in Site Reliability Engineering, HPC Engineering, DevOps, or a similar role.
- Strong understanding of modern AI infrastructure, GPU architectures, hardware performance optimization, and Linux-based distributed systems.
- Experience configuring and troubleshooting InfiniBand, RoCE, CLOS fabrics, 100GbE, Ethernet switching, GPU-direct, and NCCL environments.
- Solid Python and Go skills and experience collaborating with software engineering teams on internal tooling.
- Experience with monitoring and alerting tools such as Prometheus, Grafana, and ClickHouse.
- Proficiency with automation and configuration management tools such as Ansible and Terraform.
- Preferred experience with PyTorch, TensorFlow, DeepSpeed, MLPerf, Docker, Kubernetes, HPC resources, NVIDIA hardware and firmware, data center power and thermal design, chaos engineering, and compliance frameworks including SOC 2 and ISO 27001.
Benefits
- Generous cash and equity compensation.
- Health, dental, and vision coverage for employees and dependents.
- Wellness and commuter stipends for select roles.
- 401(k) plan with a 2% company match for USA employees.
- Flexible paid time off plan.
Categories
DevOpsSite Reliability
About Lambda
Lambda provides GPU cloud computing and on-prem AI hardware—servers, clusters, and workstations—for teams training and serving large ML models. Its products include NVIDIA H100/A100 instances, managed clusters, and the Lambda Stack software, sold via usage-based cloud pricing and hardware sales. Founded in 2012 and headquartered in San Francisco, the privately held company serves researchers, startups, enterprises, and hyperscalers.