Staff AI Observability & Telemetry Engineer
BitDeer Technologies Group2 months ago
Remote, United States or San Jose, CA, USAStaff+
Responsibilities
- Architect and scale high-cardinality telemetry infrastructure using highly available time-series databases for massive ingestion rates.
- Integrate NVIDIA DCGM, network switch telemetry, IPMI, and Redfish exporters into the Kubernetes observability stack.
- Build eBPF-based diagnostic tools for network congestion, kernel-level I/O latency, and distributed training bottlenecks.
- Develop dashboards and alerting pipelines that proactively cordon degraded hardware before customer training jobs are affected.
- Design metric pipelines for multi-tenant consumption billing based on real-time GPU and network utilization.
- Create observability standards for AI-native workloads in collaboration with GPU Systems and Scheduling teams.
- Lead technical design reviews, mentor team members, and promote best practices for high-performance telemetry collection and analysis.
Requirements
- Bachelor’s or master’s degree in Computer Science, Electrical Engineering, or a related field.
- At least 6 years of software or site reliability engineering experience.
- Deep hands-on expertise in the Prometheus and OpenTelemetry ecosystem.
- Advanced proficiency in Go and extensive experience writing custom Kubernetes metric exporters and operators.
- Hands-on experience with eBPF and BCC, plus deep Linux performance tuning experience.
- Strong familiarity with AI hardware metrics, including GPU power states, SM utilization, memory bandwidth, and high-performance network telemetry.
- Proven experience operating, debugging, and scaling large-scale telemetry stacks in high-performance computing or cloud environments.
- Strong technical leadership and cross-functional architectural influence skills.
- Excellent communication skills and the ability to translate complex system requirements into engineering milestones.
- Experience in high-velocity, high-growth engineering environments is strongly preferred.
Tech Stack
Categories
About BitDeer Technologies Group
Bitdeer Technologies Group (NASDAQ: BTDR) builds and operates Bitcoin‑mining infrastructure and AI/HPC cloud services for miners and enterprise compute customers. The Singapore‑headquartered public company, founded in 2021, designs ASIC chips and manufactures mining rigs, and runs data centers across the United States, Bhutan, Norway, Canada, Malaysia, and Ethiopia with a diversified 3 GW energy portfolio.