3 months ago
Base Salary
$267k - $356k/yr
Responsibilities
- Own reliability, performance, and capacity health for Lambda’s production storage fleet across all data centers.
- Build and maintain monitoring, dashboards, alerting, and pager routing for storage performance, capacity, and hardware failures.
- Investigate and resolve storage incidents using telemetry, logs, and performance profiling, from individual NIC failures to cluster-wide rebuilds.
- Automate ticketing, escalation, incident-response, drive replacement, node swaps, rebuild monitoring, and capacity rebalancing.
- Implement CI/CD pipelines for storage automation and tooling.
- Automate deployment and configuration of software-defined storage across new and existing sites with Storage Engineering, Fleet Orchestration, and Release Engineering.
- Diagnose low-level I/O and networking issues involving NICs, RDMA/RoCE/InfiniBand fabrics, and multipathing.
- Participate in an on-call rotation and improve MTTR through automation and root-cause remediation.
Requirements
- 5+ years of experience operating Linux systems in production or HPC environments, equivalent to at least 60 months.
- Hands-on experience operating large-scale storage on scale-out or software-defined platforms such as CEPH, Lustre, or GPFS.
- Experience integrating software-defined storage platforms with management and data-plane APIs.
- Ability to own production storage incidents from initial alert through root cause analysis and postmortem.
- Experience with monitoring and logging platforms such as Prometheus, Grafana, Alertmanager, Datadog, or SumoLogic.
- Experience with Kubernetes, GitOps tooling, and hands-on troubleshooting.
- Experience with CI/CD tooling, Docker or Podman, and systems programming in Python or Go.
- Experience with Infrastructure as Code using Terraform or Ansible.
- Understanding of file, object, block, and structured storage protocols including NFS, SMB, S3, NVMe-oF/TCP, vector databases, and SQL.
- Preferred experience with VAST, Weka, NetApp, Dell PowerScale, GPFS, Lustre, Kubernetes CSI drivers, SR-IOV, KVM/QEMU, GPUDirect Storage, RDMA, InfiniBand, RoCE, ethtool, mlxlink, clush, and open-source storage projects.
Benefits
- The role requires working from Lambda’s San Francisco or San Jose office four days per week, with Tuesday as the designated work-from-home day.
- Lambda offers cash and equity compensation, with the salary range determined by market data and candidate qualifications.
- Health, dental, and vision coverage are provided for employees and dependents.
- Wellness and commuter stipends are available for select roles.
- 401(k) plan with a 2% company match for U.S. employees.
- Flexible paid time off plan.
Tech Stack
Categories
DevOpsSite Reliability
About Lambda
Lambda provides GPU cloud computing and on-prem AI hardware—servers, clusters, and workstations—for teams training and serving large ML models. Its products include NVIDIA H100/A100 instances, managed clusters, and the Lambda Stack software, sold via usage-based cloud pricing and hardware sales. Founded in 2012 and headquartered in San Francisco, the privately held company serves researchers, startups, enterprises, and hyperscalers.