Site Reliability Engineer, AI Observability
Appnovation Technologies24 hours ago
Toronto, Canada or Montréal, CanadaSenior
Responsibilities
- Diagnose and resolve failures across ClickHouse, PostgreSQL, Redis, ingestion workers, and Kubernetes, including queue backpressure and worker drain behavior.
- Operate ClickHouse in production, including Keeper quorum, replication, shard and replica topology, and S3 storage tiering.
- Rehearse, time, document, and test backup restores against failure scenarios.
- Plan and rehearse safe upgrades with rollback plans for partially completed migrations.
- Build monitoring, alerting, and user-focused service levels from scratch.
- Maintain Kubernetes manifests, Helm values, and Argo CD applications through the delivery pipeline.
- Write and maintain incident runbooks, SOPs, and post-incident documentation.
- Own onboarding and support for internal platform users and automate recurring operational work.
- Partner with the platform engineer on platform adoption and configuration while owning migrations and rollbacks.
Requirements
- Hands-on experience with Kubernetes on AWS, including managed EKS, Helm values, and Argo CD applications.
- Hands-on production experience running ClickHouse, including replication, Keeper quorum, shard and replica topology, and backup and restore.
- Experience building monitoring, alerting, and service levels from scratch.
- Experience safely upgrading self-hosted software, including schema migrations, lower-environment rehearsals, and rollback planning.
- Operational experience with PostgreSQL and Redis sufficient to debug metadata-store and queue problems, including backpressure and worker drain.
- Strong operational writing skills for runbooks, SOPs, and post-incident reviews.
- Preferred experience with OpenTelemetry Collector pipelines, alerting design, and Grafana dashboards.
- Preferred experience with OIDC or enterprise SSO integration, GitHub Actions approval-gated pipelines, and LLM observability tools such as Langfuse, LangSmith, or Arize Phoenix.
- Experience in pharma, life sciences, regulated industries, or consulting is preferred.
Tech Stack
Categories
DevOpsSite Reliability
About Appnovation Technologies
Appnovation Technologies is a privately held digital consultancy that designs, builds, and supports customer-facing platforms for enterprises, combining strategy, UX, engineering, and managed services. Its teams deliver web and mobile development, Drupal and open-source implementations, Atlassian consulting, and data/AI-enabled solutions for large organizations. Founded in 2007 and headquartered in Vancouver, it serves global clients including Pfizer, Google, Yeti, TCL, and AstraZeneca.