3 months ago
Toronto, CanadaSenior
Responsibilities
- Design and execute prompt-based test scenarios covering happy paths, edge cases, and adversarial inputs.
- Evaluate AI outputs for accuracy, relevance, consistency, and workflow alignment.
- Build and maintain evaluation frameworks, structured test libraries, scoring rubrics, and regression suites.
- Identify and document failure modes, hallucinations, reasoning gaps, and unexpected agent behaviors.
- Partner with Product and Engineering on acceptance criteria, quality gates, and AI feature releases.
- Define quality metrics and report accuracy, failure frequency, and regression trends to stakeholders.
- Stay current on LLM evaluation, prompt engineering, and agentic testing methodologies.
Requirements
- At least 5 years of experience in QA, product operations, AI/ML evaluation, or a closely related role.
- Hands-on experience testing or evaluating LLM-powered or agentic AI products.
- Strong prompt engineering skills and understanding of how wording, context, and structure affect model behavior.
- Ability to write scripts or use evaluation tools; Python is a plus but full-stack engineering experience is not required.
- Strong analytical thinking and ability to articulate subtle AI reasoning failures.
- Clear written communication for technical and non-technical audiences.
- Familiarity with evaluation frameworks such as LangSmith, PromptFlow, or Braintrust is preferred.
- Experience with AI red-teaming or adversarial testing is preferred.
- Experience writing end-to-end tests with Playwright is preferred.
- Background in B2B SaaS or developer-tools companies and familiarity with iOS/Android testing are preferred.
Benefits
- Competitive compensation and early equity at a venture-backed company.
- Medical, dental, and vision coverage.
- Three weeks of vacation, unlimited sick and mental health days, and a company-wide end-of-year shutdown.
- $500 home-office setup stipend.
- Unlimited token usage and access to AI tools.
- High-impact environment with significant ownership and leadership opportunities.
Tech Stack
iOSPython
Categories
