Principal Engineer, Observability
Firmus Technologies3 days ago
Singapore, SingaporeStaff+
Responsibilities
- Own the multi-site observability architecture spanning telemetry collection, transport, storage, querying, alerting, APIs, and applications.
- Design the platform to scale tenfold through sharding, retention tiers, cardinality budgets, capacity planning, and load testing.
- Define and enforce telemetry schemas, naming, labeling, versioning, delivery guarantees, and alerting standards across engineering teams.
- Write production code for fleet-scale collection, telemetry pipelines, query performance, observability APIs, applications, health checks, and topology modeling.
- Build self-service dashboards, alerts, and SLOs as code while providing versioned telemetry interfaces to internal consumers, customers, and partners.
- Develop a live topology model covering sites, halls, racks, trays, GPUs, network links, and cooling systems.
- Lead architecture reviews, technical decisions, incident response, post-mortems, reliability improvements, and cost management.
- Maintain security, patching, SOC 2 Type 2, and ISO 27001 controls across new sites and platform upgrades.
- Mentor engineers, lead technical interviews, shape the roadmap with partner teams, and represent observability in customer briefings.
Requirements
- Bachelor’s degree in computer science or a related technical field, or equivalent practical experience.
- 10+ years in software, infrastructure, or platform engineering, including at least 3 years at principal level or equivalent technical leadership across multiple teams.
- Experience designing and operating metrics, logs, and traces platforms handling millions of samples per second with Prometheus-compatible stores, high cardinality, long retention, and global multi-site querying.
- Experience operating streaming telemetry ingestion using technologies such as Kafka or Redpanda with Vector or the OpenTelemetry Collector, including handling lag, backpressure, and broker failures.
- Experience defining telemetry schemas using OpenTelemetry or OpenMetrics and enabling teams to build self-service dashboards and alerts.
- Experience collecting hardware or network telemetry through Redfish, gNMI with OpenConfig models, or DCGM.
- Production experience running and upgrading stateful, high-throughput observability services on Kubernetes without data loss.
- Strong Go or Python skills and experience building production services, APIs, or applications used by other teams or customers.
- Experience implementing access control, secrets handling, patching, and production security controls under SOC 2 Type 2, ISO 27001, or an equivalent framework.
- Experience establishing alerting standards, SLOs, incident reviews, and measurable improvements to detection and restoration times.
- Clear written and verbal English communication for engineering, executive, and customer audiences.
- Willingness to participate in on-call rotations and travel overseas occasionally.
- Preferred experience includes AI cloud or high-performance computing, GPU fleet health, facilities or operational technology telemetry, digital twins, customer-facing telemetry, analytics or machine-learning pipelines, Kubernetes operators, and secure AI-assisted engineering.
- Preferred certifications include CKA, CKAD, Prometheus or OpenTelemetry Certified Associate, and Confluent Certified Administrator for Apache Kafka.
Benefits
- Full-time employment based in Singapore.
- Hybrid work arrangement with 4 days in the office and 1 day working from home.
- Occasional overseas travel is required.
- Participation in the on-call rotation for owned services.
Tech Stack
Categories
About Firmus Technologies
Firmus Technologies builds energy‑efficient AI infrastructure, developing liquid‑cooled “AI Factory” data centers and operating a large‑scale GPU cloud for model training. The company sells capacity and services to developers, enterprises, education, and government customers, with a focus on energy and cost efficiency across Asia‑Pacific. Founded in 2019 in Australia, Firmus is privately held and headquartered in St Leonards, Tasmania.