Senior Observability & Reliability Engineer
CACI International3 days ago
Base Salary
$90k - $190k/yr
Responsibilities
- Deploy, administer, secure, and maintain Grafana as the primary visualization and alerting interface.
- Build and maintain role-based dashboards for platform engineering, SRE, security operations, application teams, FinOps, and leadership.
- Design, implement, tune, and maintain alerts for availability, performance, saturation, errors, security events, telemetry gaps, storage issues, and SLO violations.
- Establish alert severity, deduplication, grouping, inhibition, routing, escalation, silencing, suppression, and maintenance-window standards.
- Define and maintain SLIs, SLOs, error budgets, service-health standards, reliability reporting, and meta-monitoring dashboards.
- Write and optimize PromQL, LogQL, TraceQL, Elastic Query DSL/KQL, and other relevant query languages.
- Build RED-method dashboards, service-level monitoring, trace-to-log correlation, and deployment-aware operational views with application teams.
- Implement dashboard, query, and administrative access controls with IAM, security, and data-governance stakeholders.
- Integrate alerts with ServiceNow, GitLab, webhooks, ticketing systems, and enterprise notification workflows.
- Maintain Grafana provisioning, GitOps, configuration-as-code, and operational support for Kubernetes or OpenShift environments.
Requirements
- Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field; equivalent experience may substitute.
- At least 7 years of experience in SRE, observability engineering, monitoring engineering, platform operations, DevOps, or production support.
- At least 3 years of advanced Grafana administration, dashboard development, and alerting experience.
- Strong experience with SLIs, SLOs, error budgets, alerting standards, incident response, and post-incident improvement processes.
- Ability to create usable dashboards and alerts from metrics, logs, traces, and event data.
- Proficiency in PromQL and/or LogQL; Elastic Query DSL/KQL experience is strongly desired.
- Experience with Kubernetes or OpenShift and cloud-native operational troubleshooting.
- Hands-on experience with Git, GitLab CI/CD, Grafana provisioning, GitOps, and configuration-as-code.
- Desired experience includes Grafana Enterprise, Elastic, Prometheus/Mimir, Loki, Tempo, OpenTelemetry, Grafana Alloy, OpenCost, and dashboards for security operations or cloud cost management.
- Familiarity with air-gapped, classified, DoD, federal, or highly regulated environments and standards including NIST SP 800-53, NIST SP 800-137, ICD 503, RMF/ATO evidence, and FIPS constraints.
- Minimum clearance required to start is TS/SCI with Polygraph.
Benefits
- Full-time regular employee position with up to 10% travel within the continental United States.
- Flexible time off benefit and access to learning and development resources.
- Healthcare, wellness, financial, retirement, family support, continuing education, and time off benefits.
- Position may be worked in more than one location.
Tech Stack
Categories
Site Reliability
About CACI International
CACI International is a public government contractor that delivers enterprise IT, cybersecurity, electronic warfare, space, and intelligence solutions to U.S. defense, intelligence, and federal civilian customers. It provides systems engineering, software and data services, and mission support across C4ISR and business systems, primarily through long-term government contracts. Founded in 1962 and headquartered in Reston, Virginia, CACI trades on the NYSE under the ticker CACI.