5 months ago
Remote, IrelandStaff+
Responsibilities
- Write production software and reliability improvements in Grafana Cloud services and the Mimir, Loki, and Tempo codebases.
- Build automation and platform tooling to reduce operational toil across regions and cloud providers.
- Author and review technical designs and shape scalable, operable feature development.
- Define and evolve per-tenant SLOs and reliability models, using incident learnings to address systemic failures.
- Improve observability and alerting so issues are detected early and escalations are actionable.
- Participate in weekday and periodic weekend on-call rotations for high-SLA customers.
- Lead customer-impacting incident response, technical escalations, customer bridge calls, and blameless post-incident reviews.
- Investigate escalations, tune capacity and configuration, follow up on incident actions, and automate recurring fixes.
- Set technical direction across squads, mentor engineers, and contribute to roadmaps and product strategy.
Requirements
- 8+ years of software engineering experience, including significant experience building and owning distributed systems in production.
- Strong production software development skills in at least one object-oriented language, with the ability to make targeted changes in large open-source codebases.
- Deep understanding of distributed-system performance, scaling, and failure modes.
- Experience operating multi-tenant systems at scale on Kubernetes in AWS, GCP, or Azure.
- Experience with infrastructure-as-code tools such as Terraform, Helm, or Jsonnet.
- Hands-on experience designing and implementing SLOs.
- Production on-call, incident-response, troubleshooting, and post-incident review experience.
- Staff-level technical leadership, including cross-team project leadership, technical direction, and mentoring.
- Working knowledge of Linux internals, networking, and cloud storage.
- Experience with Mimir, Loki, Tempo, Pyroscope, Prometheus, or other observability and time-series databases is preferred.
- Open-source contributions and customer reliability engineering or customer-facing escalation experience are preferred.
Benefits
- 100% remote work for candidates in the UK, Sweden, Spain, Germany, or Ireland.
- Weekday on-call shifts cover local working hours, with a weekend rotation approximately once every six to eight weeks.
- Global collaborative culture with autonomy, transparent communication, and empowered teams.
- Equity, bonus where applicable, and other company benefits.
- In-person onboarding.
- 30 days of annual leave per year, including three Grafana Shutdown Days, subject to local legislation.
- Career growth pathways and access to approachable, transparent leadership.
Tech Stack
Categories
Site Reliability
About Grafana
Grafana Labs builds open-source observability tools and a managed SaaS, Grafana Cloud, used by engineering and SRE teams to monitor, visualize, and analyze telemetry. Its products include Grafana dashboards plus Loki (logs), Tempo (traces), and Mimir (metrics), offered as cloud subscriptions and enterprise software. Privately held and headquartered in New York, it serves customers such as Anthropic, Bloomberg, NVIDIA, Microsoft, and Salesforce.