5 months ago
Remote, United KingdomStaff+
Responsibilities
- Write production software and reliability improvements for Grafana Cloud services and the Mimir, Loki, and Tempo codebases.
- Build automation and platform tooling to reduce operational toil across regions and cloud providers.
- Author and review technical designs and shape scalable, operable feature designs.
- Define and evolve per-tenant SLOs and reliability models using SLO burn data and incident learnings.
- Improve observability and alerting to identify customer-impacting problems earlier.
- Participate in weekday 12-hour on-call shifts and a weekend rotation approximately once every six to eight weeks.
- Lead customer-impacting incident response, technical escalations, customer bridge calls, and blameless post-incident reviews.
- Investigate escalations, tune capacity and configuration, follow up on incident actions, and turn recurring issues into permanent fixes or automation.
- Mentor engineers, teach reliability practices, drive cross-team projects, and help shape roadmaps and product strategy.
Requirements
- 8+ years of software engineering experience, including significant experience building and owning distributed systems in production.
- Strong production software development skills, ideally in at least one object-oriented language, with the ability to make targeted changes in large open-source codebases.
- Deep understanding of distributed-system performance, scaling, and failure modes.
- Experience running multi-tenant systems at scale on Kubernetes in AWS, GCP, or Azure.
- Experience with infrastructure-as-code tools such as Terraform, Helm, or Jsonnet.
- Hands-on experience designing and implementing SLOs.
- Experience being on call for production systems and participating in blameless incident response and post-incident reviews.
- Staff-level technical leadership, including cross-team project leadership, technical direction, and mentoring.
- Working knowledge of Linux internals, networking, and cloud storage.
- Experience with Mimir, Loki, Tempo, Pyroscope, Prometheus, or other observability and time-series databases is preferred.
- Open-source contributions and customer reliability engineering or customer-facing escalation experience are preferred.
Benefits
- 100% remote work for candidates in the UK, Sweden, Spain, or Germany.
- In-person onboarding is provided.
- 30 days of annual leave per year, including three Grafana Shutdown Days, subject to local legislation.
- Equity, bonus if applicable, career growth pathways, and a transparent, collaborative work culture.
- Weekday on-call shifts cover local working hours, with weekends handled by a separate rotation.
Tech Stack
Categories
BackendSite Reliability
About Grafana
Grafana Labs builds open-source observability tools and a managed SaaS, Grafana Cloud, used by engineering and SRE teams to monitor, visualize, and analyze telemetry. Its products include Grafana dashboards plus Loki (logs), Tempo (traces), and Mimir (metrics), offered as cloud subscriptions and enterprise software. Privately held and headquartered in New York, it serves customers such as Anthropic, Bloomberg, NVIDIA, Microsoft, and Salesforce.