5 months ago
Base Salary
$175k - $210k/yr
Responsibilities
- Lead multi-quarter technical initiatives from problem framing through rollout, including trace aggregation APIs, autoscaling cells, customer limits, and query engine improvements.
- Own the architecture of Tempo’s ingestion, storage, query, and metrics-generation components.
- Design structured, deterministic, and discoverable APIs for products, LLM-driven assistants, and external integrators.
- Drive operational excellence through SLO ownership, automation, parameterized rollouts, actionable alerts, and toil reduction.
- Improve performance for high-volume ingestion and long query ranges.
- Partner with Product and sibling teams including App Observability, Asserts, Drilldown, and Grafana Assistant.
- Mentor engineers through code reviews, design feedback, pairing, and technical writing.
- Participate in on-call and incident response for services built by the team.
- Contribute to the Tempo open-source community, including reviewing external contributions and steering the project publicly.
Requirements
- Track record of leading complex, multi-quarter initiatives spanning design, delivery, and operations.
- Substantial hands-on experience building and operating distributed data systems in production, such as ingestion pipelines, storage engines, or query execution systems.
- Strong software craftsmanship with the ability to write clean, robust, performant, maintainable software.
- Strong Go experience or a path to Go; deep experience with Rust, C, or C++ is considered transferable.
- Experience owning production services, participating in on-call, reducing toil, and treating SLOs as product features.
- Customer-focused and pragmatic approach using short feedback loops and iterative delivery.
- Clear communication and leadership through design documents, reviews, and shipped code in a remote asynchronous environment.
- Bonus: experience with tracing, OpenTelemetry, or large-scale observability systems.
- Bonus: experience designing query languages, SQL- or TraceQL-like engines, or programmatically consumed APIs.
- Bonus: experience with Parquet or purpose-built on-disk formats for analytical workloads.
- Bonus: experience operating multi-tenant, multi-cell SaaS infrastructure at scale on Kubernetes.
- Bonus: experience building structured APIs, metadata or discovery endpoints, deterministic outputs, or evaluation harnesses for AI/LLM consumers.
- Bonus: open-source contribution or maintainership experience.
- Bonus: experience using Grafana, Prometheus, Loki, or Tempo on call or in a homelab.
- Bonus: experience working in a fully remote, globally distributed team.
Benefits
- 100% remote position for candidates in the US and Canada
- Remote-first, asynchronous work with regular video meetings
- Company-funded usage budget for AI coding assistants
- Access to frontier models including GPT-Codex, Claude Opus, and Gemini
- Restricted Stock Units (RSUs) included with all roles
- 30 days of annual leave per year, including 3 Grafana Shutdown Days
- In-person onboarding
About Grafana
Grafana Labs builds open-source observability tools and a managed SaaS, Grafana Cloud, used by engineering and SRE teams to monitor, visualize, and analyze telemetry. Its products include Grafana dashboards plus Loki (logs), Tempo (traces), and Mimir (metrics), offered as cloud subscriptions and enterprise software. Privately held and headquartered in New York, it serves customers such as Anthropic, Bloomberg, NVIDIA, Microsoft, and Salesforce.