1 day ago
Toronto, CanadaEntry Level
Responsibilities
- Design and implement intelligent unit, integration, and end-to-end testing systems for agentic AI applications.
- Own test specifications for LLM-powered applications and architect prompt validation and observability frameworks.
- Build adaptive and self-healing test solutions using Playwright, Stagehand, Locust, Allure, and LLM orchestration frameworks.
- Develop SDET agents that generate test cases, validate prompts, and autonomously triage defects.
- Lead autonomous test execution, AI-driven defect detection and classification, LLM-powered analysis, and test observability.
- Partner with AI/ML engineers, prompt engineers, developers, platform engineers, and business stakeholders to improve reliability and performance.
- Document LLM behavior test cases and agent interaction patterns and maintain a knowledge base for AI testing.
Requirements
- At least 3 years of quality engineering experience across cloud, distributed systems, APIs, databases, and AI/ML systems.
- 3–5 years of hands-on automation experience covering component, integration, and end-to-end scenarios.
- Deep experience testing LLM-based agents, multi-agent systems, prompt validation, output consistency, and non-deterministic behavior.
- Hands-on experience with LangChain, LangGraph, CrewAI, AutoGen, or similar agent workflow frameworks.
- Experience with CI/CD, GitHub Actions, JIRA, qTest, and integrating AI model evaluation into deployment pipelines.
- Expertise with Playwright, Locust, Python httpx, and AI evaluation tools such as promptfoo.
- Programming experience in Python, TypeScript, SQL, and Git, including asynchronous programming for agent orchestration.
- Experience defining AI test strategies, evaluating LLM outputs, reviewing agent code and prompt templates, and working in Agile or iterative development environments.
- Knowledge of prompt engineering techniques such as few-shot learning, chain-of-thought prompting, ReAct patterns, and prompt robustness validation is preferred.
- Knowledge of Langfuse or Arize Phoenix, RAG pipelines, vector database testing, embedding quality, and retrieval accuracy is preferred.
Benefits
- Comprehensive total rewards program with bonuses and flexible benefits.
- Competitive compensation, commissions, and stock where applicable.
- Development support through coaching and management opportunities.
- Opportunity to make a lasting impact in a dynamic, collaborative, progressive, high-performing team.
- World-class training program in financial services.
- Flexible work/life balance options.
- Full-time regular salaried employment in Toronto, Canada, with 37.5 work hours per week.
Tech Stack
Categories
About RBC
Royal Bank of Canada (RBC) provides personal and commercial banking, payments, mortgages, investing, wealth management, insurance, and capital markets services to consumers, businesses, and institutions. The public company, headquartered in Toronto and founded in 1864, serves more than 17 million clients across Canada, the U.S., and other global markets. Revenue comes from interest income, fees, advisory, and trading across its diversified financial services businesses, and it is Canada’s largest bank by market capitalization.