3 months ago
Base Salary
$150k - $220k/yr
Responsibilities
- Own versioned OS images, kernel tuning, GPU and NIC driver stacks, and automated pipelines for production-ready GPU nodes.
- Build node acceptance, burn-in, bandwidth, topology, and performance validation suites before nodes enter scheduler pools.
- Execute rolling kernel, driver, and toolkit upgrades while enforcing fleet consistency and detecting configuration drift.
- Automate unhealthy-node detection and cordon, drain, reboot, re-image, and hardware-repair handoff workflows.
- Manage cluster configuration with Ansible or SaltStack, Git-based review, CI validation, and canary rollouts.
- Build reproducible provisioning and image pipelines using PXE, MaaS, Packer, or similar tooling.
- Operate GPU-enabled Kubernetes for inference and Slurm or Run:AI for distributed training workloads.
- Maintain container images, registries, NVIDIA Container Toolkit or ROCm container stacks, and optimized PyTorch and JAX environments.
- Deploy and debug NVIDIA CUDA and AMD ROCm driver and runtime stacks, kernel modules, GPUDirect, and RDMA software.
- Tune NCCL and RCCL across NVLink, NVSwitch, InfiniBand, and RoCE fabrics and maintain topology-aware job placement.
- Investigate GPU-cluster failures including NCCL hangs, CUDA memory leaks, ROCm crashes, straggler nodes, and throughput regressions.
- Own software-layer monitoring, GPU utilization reporting, and cluster-efficiency observability.
- Support foundation-model training and inference teams with distributed job debugging, checkpoint I/O optimization, and serving-stack performance tuning.
Requirements
- At least 5 years of systems or infrastructure engineering experience with significant GPU cluster, HPC, or large-scale ML infrastructure experience.
- Bachelor's or master's degree in Computer Science, Computer Engineering, Electrical Engineering, or a related technical field.
- Deep Linux internals expertise, including kernel modules, DKMS, systemd, cgroups, NUMA, and system performance tuning.
- Hands-on NVIDIA CUDA and/or AMD ROCm driver and runtime experience on modern accelerators, including kernel-level debugging.
- Production Kubernetes experience with GPU workloads and working knowledge of Slurm or Run:AI, or the reverse combination.
- Strong Ansible or SaltStack configuration-management experience with Git-based, code-reviewed infrastructure workflows.
- Experience with Packer, MaaS, Foreman, Terraform, or similar provisioning and image tooling.
- Client-side experience with Lustre, GPFS, or Weka distributed filesystems and checkpoint I/O optimization.
- Container experience with Docker or containerd and NVIDIA Container Toolkit or a ROCm equivalent.
- Proficiency in Python and Bash.
- Working knowledge of NCCL, RDMA networking, InfiniBand or RoCE, GPUDirect, and PyTorch or JAX runtime behavior.
- Preferred experience supporting foundation-model training teams, debugging multi-node failures, tuning checkpoint pipelines, or triaging framework performance.
- Preferred experience deploying and tuning vLLM, Triton Inference Server, or TensorRT-LLM for latency and throughput.
- Preferred experience with Nsight Systems, Nsight Compute, rocprof, perf, or eBPF.
Benefits
- Medical, dental, and vision insurance.
- 401(k) plan.
- Daily lunch, snacks, and beverages.
- Flexible time off.
- Competitive salary and equity.
Categories
