5 months ago
Remote, United Kingdom +3 moreStaff+
Responsibilities
- Write production software, automation, and product changes that improve database resilience, self-healing, and scalability.
- Build platform tooling to reduce operational toil across regions and cloud providers.
- Author and review technical designs and ensure features are scalable and operable.
- Define and evolve per-tenant SLOs and reliability models, using incident learnings to address systemic failures.
- Improve observability and alerting so issues are detected early and escalations are actionable.
- Participate in weekday and periodic weekend on-call rotations for high-SLA customers.
- Lead complex customer-impacting incident response, customer bridge calls, and blameless post-incident reviews.
- Investigate escalations, tune capacity and configuration, follow up on incident actions, and automate recurring fixes.
- Mentor engineers, promote reliability practices, and help shape cross-team roadmaps and product strategy.
Requirements
- 8+ years of software engineering experience, including significant experience building and owning distributed systems in production.
- Strong production software development skills, ideally in at least one object-oriented language, including the ability to modify large open-source codebases.
- Deep understanding of distributed-system performance, scaling, and failure modes.
- Experience operating multi-tenant systems at scale on Kubernetes in AWS, GCP, or Azure.
- Experience with infrastructure-as-code tools such as Terraform, Helm, or Jsonnet.
- Hands-on experience designing and implementing SLOs.
- Production on-call and incident-response experience, including writing post-incident reviews.
- Staff-level technical leadership through cross-team projects, technical direction, and mentoring.
- Working knowledge of Linux internals, networking, and cloud storage.
- Experience with Mimir, Loki, Tempo, Pyroscope, Prometheus, observability, or time-series databases is a plus.
- Open-source contributions and customer reliability engineering or customer-facing escalation experience are a plus.
Benefits
- 100% remote role for candidates in the UK, Sweden, Spain, or Germany.
- Weekday on-call shifts cover local working hours, with a weekend rotation approximately once every six to eight weeks.
- Equity, bonus where applicable, and other benefits.
- 30 days of annual leave per year, including three Grafana Shutdown Days, subject to local legislation.
- In-person onboarding.
- Career growth pathways, transparent communication, high-trust teams, and a global collaborative culture.
Tech Stack
Categories
Site Reliability
About Grafana
Grafana Labs builds open-source observability tools and a managed SaaS, Grafana Cloud, used by engineering and SRE teams to monitor, visualize, and analyze telemetry. Its products include Grafana dashboards plus Loki (logs), Tempo (traces), and Mimir (metrics), offered as cloud subscriptions and enterprise software. Privately held and headquartered in New York, it serves customers such as Anthropic, Bloomberg, NVIDIA, Microsoft, and Salesforce.