Senior Observability & Reliability Engineer
CACI International3 days ago
Base Salary
$90k - $190k/yr
Responsibilities
- Deploy, administer, secure, and maintain Grafana as the primary visualization and alerting interface.
- Build role-based dashboards for platform engineering, SRE, security operations, application teams, FinOps, and leadership.
- Design, tune, and maintain alerts for availability, performance, saturation, errors, security events, telemetry gaps, storage issues, and SLO violations.
- Establish alert severity, deduplication, grouping, inhibition, routing, escalation, silencing, suppression, and maintenance-window standards.
- Define and maintain SLIs, SLOs, error budgets, service-health standards, reliability reporting, and operational-readiness practices.
- Build meta-monitoring dashboards for telemetry pipelines, query latency, ingestion, storage, node health, alerting, and identity-provider health.
- Write and optimize PromQL, LogQL, TraceQL, Elastic Query DSL/KQL, and other approved query languages.
- Partner with application teams on RED-method dashboards, service-level monitoring, trace-to-log correlation, and deployment-aware operational views.
- Implement dashboard, query, and administrative access controls with IAM, security, and data-governance stakeholders.
- Integrate alerts with ServiceNow, GitLab, webhooks, ticketing systems, and enterprise notification workflows.
Requirements
- Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field, or equivalent experience.
- At least 7 years of experience in SRE, observability engineering, monitoring engineering, platform operations, DevOps, or production support.
- At least 3 years of advanced Grafana administration, dashboard development, and alerting experience.
- Strong experience with SLIs, SLOs, error budgets, alerting standards, incident response, and post-incident improvement.
- Ability to create usable dashboards and alerts from metrics, logs, traces, and event data.
- Proficiency in PromQL and/or LogQL; Elastic Query DSL/KQL experience is strongly desired.
- Experience with Kubernetes or OpenShift and cloud-native operational troubleshooting.
- Hands-on experience with Git, GitLab CI/CD, Grafana provisioning, GitOps, and configuration-as-code.
- Desired experience includes Grafana Enterprise, Elastic, Prometheus/Mimir, Loki, Tempo, OpenTelemetry, Grafana Alloy, OpenCost, security-operations dashboards, and cloud-cost management.
- Experience in air-gapped, classified, DoD, federal, or highly regulated environments and familiarity with NIST SP 800-53, NIST SP 800-137, ICD 503, RMF/ATO evidence, and FIPS constraints are desired.
- TS/SCI clearance with polygraph is required to start.
Benefits
- Flexible time off, healthcare, wellness, financial, retirement, family support, continuing education, learning and development, and other time-off benefits are offered.
- The position is full time and regular, with up to 10% travel within the Continental US.
- The role may be worked in more than one location.
- The position supports classified and regulated-environment work and requires TS/SCI with polygraph clearance.
Tech Stack
Categories
DevOpsSite Reliability
About CACI International
CACI International is a public government contractor that delivers enterprise IT, cybersecurity, electronic warfare, space, and intelligence solutions to U.S. defense, intelligence, and federal civilian customers. It provides systems engineering, software and data services, and mission support across C4ISR and business systems, primarily through long-term government contracts. Founded in 1962 and headquartered in Reston, Virginia, CACI trades on the NYSE under the ticker CACI.