Staff AI Observability & Telemetry Engineer

1 month ago

Singapore, SingaporeStaff+

Responsibilities

Architect and scale high-cardinality telemetry infrastructure using highly available time-series databases.
Integrate NVIDIA DCGM, network switch telemetry, and IPMI/Redfish exporters into the Kubernetes observability stack.
Build eBPF-based tools to diagnose network congestion, kernel-level I/O latency, and distributed training bottlenecks.
Develop dashboards and alerting pipelines that proactively cordon degraded hardware.
Design metric pipelines for multi-tenant billing based on real-time GPU and network utilization.
Collaborate with GPU Systems and Scheduling teams on observability standards for AI-native workloads.
Lead observability architecture reviews and mentor team members on high-performance telemetry collection and analysis.

Bachelor’s or master’s degree in Computer Science, Electrical Engineering, or a related field.
At least 6 years of software or site reliability engineering experience.
Deep hands-on expertise with the Prometheus and OpenTelemetry ecosystem.
Advanced proficiency in Go and extensive experience writing custom Kubernetes metric exporters and operators.
Hands-on experience with eBPF, BCC, and Linux performance tuning.
Familiarity with AI hardware metrics, including GPU power states, SM utilization, and memory bandwidth.
Experience with high-performance network telemetry.
Demonstrated experience operating, debugging, and scaling large-scale telemetry stacks in HPC or cloud environments.
Strong technical leadership, cross-functional collaboration, and communication skills.
Experience in high-velocity, high-growth engineering environments is preferred.

Inclusive and diverse workplace culture with open workspaces and a startup environment.
Opportunity to work on new projects and contribute to the future of the digital asset industry.
Networking opportunities with industry pioneers and enthusiasts.
Personal accountability, autonomy, fast growth, and learning opportunities.
Welfare benefits and professional development opportunities including training and mentoring.

201-500 employees