5 months ago
Remote, CanadaStaff+
Responsibilities
- Lead multi-quarter technical initiatives from problem framing through rollout, including trace aggregation APIs, autoscaling cells, customer limits, and query-engine improvements.
- Own the architecture of Tempo’s ingestion, storage, query, and metrics-generation components.
- Design structured, deterministic, discoverable APIs for Grafana products, LLM-driven assistants, and external integrators.
- Drive operational excellence through SLO ownership, automation, parameterized rollouts, actionable alerts, and toil reduction.
- Improve query latency, ingestion performance, autoscaling, multi-cell operations, and support for bursty high-cardinality workloads.
- Partner with Product, App Observability, Asserts, Drilldown, and Grafana Assistant teams to deliver platform capabilities.
- Mentor engineers through code review, design feedback, pairing, and technical writing.
- Participate in on-call and incident response for the services developed.
- Contribute to Tempo as an open-source project, including community engagement and review of external contributions.
Requirements
- Track record of leading complex, multi-quarter initiatives spanning design, delivery, and operations.
- Substantial hands-on experience building and operating distributed data systems in production, such as ingestion pipelines, storage engines, or query execution systems.
- Strong software craftsmanship and experience writing clean, robust, performant, maintainable software.
- Strong Go experience, or deep experience with systems languages such as Rust, C, or C++.
- Experience owning production services, participating in on-call rotations, reducing toil, and working with SLOs.
- Ability to lead through design documents, code reviews, collaboration, and shipped code in a fully remote, asynchronous environment.
- Customer-focused, pragmatic approach based on short feedback loops and iterative delivery.
- Bonus: experience with tracing, OpenTelemetry, observability systems, query languages, SQL- or TraceQL-like engines, or programmatic APIs.
- Bonus: experience with Parquet or purpose-built on-disk analytical storage formats.
- Bonus: experience operating multi-tenant, multi-cell SaaS infrastructure at scale on Kubernetes.
- Bonus: experience building structured, deterministic APIs and evaluation harnesses for AI/LLM consumers.
- Bonus: open-source contribution or maintainership and experience engaging technical communities.
- Bonus: prior use of Grafana, Prometheus, Loki, or Tempo in an on-call role or homelab.
- Bonus: experience working in fully remote, globally distributed teams.
Benefits
- Remote position for candidates in the US and Canada.
- 100% remote, globally distributed work with asynchronous collaboration and regular video meetings.
- In-person onboarding.
- 30 days of annual leave per year, including 3 Grafana Shutdown Days, subject to local legislation.
- Restricted Stock Units (RSUs) are included with all roles.
- Career growth pathways and a high-trust, transparent culture.
About Grafana
Grafana Labs builds open-source observability tools and a managed SaaS, Grafana Cloud, used by engineering and SRE teams to monitor, visualize, and analyze telemetry. Its products include Grafana dashboards plus Loki (logs), Tempo (traces), and Mimir (metrics), offered as cloud subscriptions and enterprise software. Privately held and headquartered in New York, it serves customers such as Anthropic, Bloomberg, NVIDIA, Microsoft, and Salesforce.