Senior AI Engineer for Observability
BMO Financial Group17 hours ago
Toronto, CanadaSenior
Base Salary
$80k - $175k/yr
Responsibilities
- Design, build, and maintain systems that track, evaluate, debug, and govern non-deterministic AI models and agent workflows.
- Trace LLM calls, agent trajectories, spans, prompts, tool calls, and intermediate outputs in production.
- Monitor model drift, data quality degradation, latency, error rates, token usage, inference costs, and log retention.
- Build automated evaluation pipelines for relevance, groundedness, hallucination detection, and prompt adherence.
- Provision and standardize observability pipelines, logging sidecars, and centralized dashboards.
- Define operational boundaries and guardrails for autonomous agents, including self-remediation rules.
- Reduce alert fatigue through intelligent anomaly detection, runbooks, and incident-response practices.
- Design, develop, train, retrain, test, and optimize machine learning and deep learning systems and frameworks.
- Perform statistical analysis, interpret test results, and provide specialized analytical and technical support.
Requirements
- Bachelor's degree in Computer Science, Engineering, or another relevant domain.
- 7–10 years of experience in Site Reliability Engineering.
- Experience in artificial intelligence observability engineering and automating detection, investigation, and remediation.
- Exposure to platform engineering and building AI capabilities.
- High proficiency in automation, monitoring, observability, buildouts, and AI engineering.
- Understanding of model architectures, evaluation metrics, token usage, and concept or data drift.
- Experience using Python for custom evaluations and SQL, LogQL, or PromQL for telemetry, log, and metrics queries.
- Experience with observability pipelines, logging sidecars, centralized dashboards, SLIs/SLOs, alert management, and root cause analysis.
- Experience implementing OpenTelemetry-based agent and LLM tracing frameworks.
- Exposure to heavily regulated industry organizations.
Benefits
- Hybrid work arrangement.
- Health insurance, tuition reimbursement, accident and life insurance, and retirement savings plans.
- Training, coaching, manager support, and network-building opportunities.
- Inclusive, equitable, and accessible workplace with accommodations available on request.
Categories
ML EngineeringSite Reliability
About BMO Financial Group
BMO Financial Group is a publicly traded North American bank offering personal and commercial banking, wealth management, and capital markets services to consumers, businesses, and institutions. Founded in 1817 and headquartered in Toronto, it operates in Canada and the United States, including BMO Bank N.A. in the U.S. The company is listed on the NYSE and TSX under the symbol BMO.