Site Reliability Engineer, AI Observability
Appnovation Technologies24 hours ago
Miami, FL, USA +3 moreSenior
Responsibilities
- Diagnose and resolve failures across ClickHouse, PostgreSQL, Redis, ingestion workers, and the Kubernetes platform.
- Operate ClickHouse, including Keeper quorum, replication, shard and replica topology, and S3 storage tiering.
- Rehearse, measure, document, and test backup restoration against failure scenarios.
- Plan and rehearse upgrades with rollback plans for partially completed migrations.
- Build monitoring, alerting, and user-focused service levels from scratch.
- Maintain Kubernetes manifests, Helm values, and Argo CD applications through the delivery pipeline.
- Write and maintain incident runbooks, SOPs, and post-incident documentation.
- Own onboarding, support, and triage for internal teams using the platform.
- Automate recurring operational work and partner with the platform engineer on migrations and rollbacks.
Requirements
- Hands-on experience with Kubernetes on AWS, including managed EKS, Helm values, and Argo CD applications.
- Production experience running ClickHouse, including replication, Keeper quorum, shard and replica topology, backup and restore, ideally with an operator.
- Experience building monitoring and alerting, including service levels that reflect user experience.
- Experience upgrading self-hosted software safely with schema-migration rehearsal and rollback planning.
- Operational experience with PostgreSQL and Redis, including debugging metadata-store, queue, backpressure, and worker-drain issues.
- Strong ability to write runbooks, SOPs, and post-incident reviews that colleagues can follow independently.
- Preferred experience with OpenTelemetry Collector pipelines, alerting design, and Grafana dashboards.
- Preferred experience with OIDC or enterprise SSO, GitHub Actions approval-gated pipelines, and LLM observability tools such as Langfuse, LangSmith, or Arize Phoenix.
- Experience in pharma, life sciences, regulated industries, consulting, or working within client teams is preferred.
Tech Stack
Categories
DevOpsSite Reliability
About Appnovation Technologies
Appnovation Technologies is a privately held digital consultancy that designs, builds, and supports customer-facing platforms for enterprises, combining strategy, UX, engineering, and managed services. Its teams deliver web and mobile development, Drupal and open-source implementations, Atlassian consulting, and data/AI-enabled solutions for large organizations. Founded in 2007 and headquartered in Vancouver, it serves global clients including Pfizer, Google, Yeti, TCL, and AstraZeneca.