Responsibilities
- Build an ingestion path that processes production-volume agent traces.
- Score agent behavior in real time rather than through batch processing.
- Enforce quality and safety guardrails in the live request path within latency and cost constraints.
- Correlate spans across services so requests form unified traces.
- Convert production traces into datasets and test cases for evaluating future agent versions.
- Set technical direction for the runtime agent-evaluation field and contribute through open source and conference talks.
- Interact frequently with RBC partners and employees across Canada and worldwide.
Requirements
- Have 8+ years of software or platform engineering experience, including 5+ years in SRE, real-time data infrastructure, observability, or large-scale stream processing.
- Demonstrate hands-on experience running high-volume telemetry systems in production, including ingestion, storage, and querying at scale.
- Understand distributed tracing and OpenTelemetry, including semantic conventions, collector configuration, and cross-service span correlation.
- Have experience routing live traffic using weighted load balancing, canary rollouts, or multi-armed-bandit routing.
- Have experience turning streaming telemetry into real-time decisions through scoring, anomaly detection, or rule and threshold evaluation.
- Have experience with an LLM observability platform such as Langfuse, MLflow, or an equivalent, and with trace-to-evaluation feedback loops.
- Preferred experience includes live request-path latency and backpressure trade-offs involving collectors, proxies, or sidecars.
- Preferred experience includes regulated industries, AI security controls, AI governance, model audit logging, runtime drift detection, open-source contributions, or published work in observability, tracing, or LLM evaluation.
Benefits
- Comprehensive total rewards program with bonuses and flexible benefits, plus competitive compensation, commissions, and stock where applicable.
- Development support through coaching and management opportunities.
- Opportunity to make a difference and lasting impact.
- Dynamic, collaborative, progressive, and high-performing team environment.
- World-class training program in financial services.
- Opportunities to do challenging work.
- Full-time regular salaried position in Toronto, Canada, with 37.5 work hours per week.
Tech Stack
Categories
About RBC
Royal Bank of Canada is a global financial institution with a purpose-driven, principles-led approach to delivering leading performance. Our success comes from the 94,000+ employees who leverage their imaginations and insights to bring our vision, values and strategy to life so we can help our clients thrive and communities prosper. As Canada's biggest bank and one of the largest in the world, based on market capitalization, we have a diversified business model with a focus on innovation and providing exceptional experiences to our more than 17 million clients in Canada, the U.S. and 27 other countries. Learn more at rbc.com. We are proud to support a broad range of community initiatives through donations, community investments and employee volunteer activities. See how at www.rbc.com/community-social-impact. http://rbc.com/legalstuff. La Banque Royale du Canada est une institution financière mondiale définie par sa raison d'être, guidée par des principes et orientée vers l'excellence en matière de rendement. Notre succès est attribuable aux quelque 94 000+ employés qui mettent à profit leur créativité et leur savoir faire pour concrétiser notre vision, nos valeurs et notre stratégie afin que nous puissions contribuer à la prospérité de nos clients et au dynamisme des collectivités. Selon la capitalisation boursière, nous sommes la plus importante banque du Canada et l'une des plus grandes banques du monde. Nous avons adopté un modèle d'affaires diversifié axé sur l'innovation et l'offre d'expériences exceptionnelles à nos plus de 17 millions de clients au Canada, aux États Unis et dans 27 autres pays. Pour en savoir plus, visitez le site rbc.com/francais Nous sommes fiers d'appuyer une grande diversité d'initiatives communautaires par des dons, des investissements dans la collectivité et le travail bénévole de nos employés. Pour de plus amples renseignements, visitez le site www.rbc.com/collectivite-impact-social. https://www.rbc.com/conditions-dutilisation/