Senior Platform Reliability Engineer, Fabric and Interconnect
Firmus Technologies2 months ago
Responsibilities
- Operate, automate, and continuously improve GPU interconnect and network fabrics, including NVLink, NVSwitch, InfiniBand, and Spectrum-X domains.
- Build guarded automation and remediation tooling for fabric fault isolation, reconvergence, and software-driven AI cluster recovery.
- Diagnose and tune performance across collective communication, interconnect, link, routing, and congestion-control layers.
- Operate DPU-based host networking, including offload paths and driver and firmware compatibility.
- Plan and execute firmware upgrade waves, fabric expansions, capacity changes, production windows, canary rollouts, verification, and rollback.
- Provide deep technical diagnosis for fabric incidents, correlate failures to physical links, and drive permanent fixes.
- Lead engineering-level vendor escalations and reproduce faults to vendor standards.
- Lead major fabric incident recovery, participate in the follow-the-sun on-call roster, mentor engineers, and maintain runbooks and performance documentation.
Requirements
- 8+ years of experience, including substantial ownership of production network or interconnect infrastructure in a 24/7 environment.
- Deep operational experience with GPU interconnect fabrics such as NVLink and NVSwitch, including topology and failure diagnosis.
- Extensive experience with InfiniBand or RoCE-based Ethernet such as NVIDIA Spectrum-X, including routing internals and congestion-control tuning.
- Experience with DPU or SmartNIC-based host networking, offload paths, and driver and firmware coordination.
- Experience planning and executing production fabric firmware upgrades and capacity expansions with defined rollback procedures.
- Strong infrastructure automation and infrastructure-as-code skills using peer review and progressive rollout practices.
- Practical scripting or programming experience with Python, Go, or Bash.
- Experience with major incident response, on-call participation, engineering-level vendor escalation, and production runbook creation.
- Preferred experience with large-scale distributed training or inference fabrics, NVIDIA rack-scale or multi-node GPU systems, cloud or colocation environments, data-center hardware fundamentals, cabling and optics, firmware management, hardware fault workflows, or relevant vendor certification.
- Bachelor’s degree in computer science, engineering, or a related discipline, or an equivalent combination of relevant experience and training.
Benefits
- Full-time employment.
- Hybrid working model; candidates must be based in the San Francisco Bay Area or willing to relocate.
- Opportunity to work with founders and experts in AI infrastructure, energy systems, and next-generation compute.
- Ownership and exposure across Firmus’s large-scale AI infrastructure roadmap.
About Firmus Technologies
Firmus Technologies builds energy‑efficient AI infrastructure, developing liquid‑cooled “AI Factory” data centers and operating a large‑scale GPU cloud for model training. The company sells capacity and services to developers, enterprises, education, and government customers, with a focus on energy and cost efficiency across Asia‑Pacific. Founded in 2019 in Australia, Firmus is privately held and headquartered in St Leonards, Tasmania.