6 days ago
Responsibilities
- Own monitoring architecture, alert suppression, and signal quality across GPU, host, and facility telemetry.
- Provide technical SEV incident command, bridge coordination, paging, escalation, and incident timeline and severity management.
- Run blameless postmortems and drive corrective actions to completion.
- Gate fleet-scale firmware, BIOS, BMC, power, and cooling changes through staged rollout and change-management processes.
- Lead reliability projects across compute, networking, storage, power, cooling, and facility systems, including cross-boundary failure diagnosis.
- Use fleet failure data and reliability metrics to develop monitoring, stop-rules, and vendor RCA inputs.
- Build playbooks, runbooks, dependency maps, and game days in partnership with the NOC.
- Define error budgets and availability objectives at campus and service boundaries.
- Participate in on-call rotations and SEV-class incident response at the Memphis/Southaven campus.
Requirements
- Bachelor's degree in Systems Engineering, Computer Science, Electrical Engineering, or a related field, or equivalent experience.
- At least 5 years of experience in site reliability, systems engineering, plant operations, or large-scale production operations.
- Large-scale incident command experience and calm technical leadership on an incident bridge or plant control room.
- Fleet- or campus-scale monitoring and observability design experience, including alert hygiene, suppression, and signal quality.
- Hands-on experience with Prometheus/Grafana, NVIDIA DCGM, or industrial telemetry and controls such as SCADA, BMS/BAS, EPMS, DCIM, PLCs, Modbus, or BACnet.
- Experience spanning at least two of compute, network, storage, power, and cooling/facilities telemetry.
- Experience writing and operating playbooks, runbooks, or SOP/MOP/EOPs with a 24/7 operations, control-room, or NOC partner.
- Scripting proficiency in Python or Bash and general experience with a systems language such as C, C++, Java, Go, or Rust.
- Preferred experience in power plants, nuclear power plants, data centers, industrial control environments, AI/ML infrastructure, supercomputing, fleet firmware and BMC management, reliability engineering, CMMS, commissioning, SLOs/SLIs, error budgets, game days, and dependency mapping.
- Ability to collaborate with NOC, data center operations, hardware failure analysis, and infrastructure engineering teams.
About xAI
Understand the Universe. We are a team of AI technologists and business leaders on a mission to build AI systems that can help humanity understand the world better. https://x.ai/careers