Staff AI Observability & Telemetry Engineer

14 days ago

Remote, United States or San Jose, CA, USAStaff+

Responsibilities

Architect and scale high-cardinality telemetry infrastructure using highly available time-series databases for massive ingestion rates.
Integrate NVIDIA DCGM, network switch telemetry, IPMI, and Redfish exporters into the Kubernetes observability stack.
Build eBPF-based diagnostic tools for network congestion, kernel-level I/O latency, and distributed training bottlenecks.
Develop dashboards and alerting pipelines that proactively cordon degraded hardware before customer training jobs are affected.
Design metric pipelines for multi-tenant consumption billing based on real-time GPU and network utilization.
Create observability standards for AI-native workloads in collaboration with GPU Systems and Scheduling teams.
Lead technical design reviews, mentor team members, and promote best practices for high-performance telemetry collection and analysis.

Bachelor’s or master’s degree in Computer Science, Electrical Engineering, or a related field.
At least 6 years of software or site reliability engineering experience.
Deep hands-on expertise in the Prometheus and OpenTelemetry ecosystem.
Advanced proficiency in Go and extensive experience writing custom Kubernetes metric exporters and operators.
Hands-on experience with eBPF and BCC, plus deep Linux performance tuning experience.
Strong familiarity with AI hardware metrics, including GPU power states, SM utilization, memory bandwidth, and high-performance network telemetry.
Proven experience operating, debugging, and scaling large-scale telemetry stacks in high-performance computing or cloud environments.
Strong technical leadership and cross-functional architectural influence skills.
Excellent communication skills and the ability to translate complex system requirements into engineering milestones.
Experience in high-velocity, high-growth engineering environments is strongly preferred.

201-500 employees