1 month ago
Remote, CanadaSenior
Responsibilities
- Review requirements specifications and technical design documents and provide feedback.
- Create, maintain, and execute comprehensive test plans and test cases.
- Design, develop, and execute automated testing scripts using TypeScript, Playwright, or comparable frameworks.
- Use AI coding assistants and agentic tools to author tests, maintain automation, and triage defects.
- Build and maintain reusable AI skills and agentic workflows for test generation, regression checks, and defect documentation.
- Apply spec-driven development practices to guide AI-assisted test generation and validation.
- Design and run evaluations for AI-powered features using defined datasets and quality metrics.
- Apply non-deterministic testing strategies to AI/LLM-based features using sampling, statistical thresholds, and repeated-run analysis.
- Configure and monitor AI observability and evaluation platforms to track model and agent performance and detect regressions.
- Identify, document, track, and close bugs and perform regression testing after fixes.
- Estimate, prioritize, plan, and coordinate testing activities.
- Lead or assist junior QA Analysts and share knowledge through documentation and presentations.
Requirements
- Proven experience in manual and automated QA testing of web applications.
- Bachelor's degree in a technical field and 5+ years of experience, or equivalent job experience.
- Proficiency in TypeScript or JavaScript and hands-on experience with Playwright or a comparable automation framework such as Cypress, Selenium, or WebdriverIO.
- Practical experience using AI coding assistants and agentic tools for test automation.
- Working understanding of spec-driven development methodologies such as AI-DLC.
- Ability to build and maintain custom AI skills, prompts, or agent workflows.
- Understanding of non-deterministic testing for AI/LLM outputs.
- Experience designing evaluation datasets and metrics including accuracy, relevance, hallucination, toxicity, and latency.
- Familiarity with AI observability and evaluation-monitoring platforms such as Arize AX/Phoenix, LangSmith, or Braintrust.
- Solid grounding in QA methodologies, the SDLC, test planning, and estimation.
- Strong analytical, written, and verbal communication skills, with the ability to mentor junior QA Analysts.
Benefits
- Fully remote-first work with flexible work arrangements.
- Health and wellness benefits, retirement savings programs, bonus-plan eligibility, paid time off, and medical and dental benefits based on country of location.
- New hire equipment allowance and monthly flex allowance.
- Career growth and internal mobility opportunities.
- Employee-driven diversity, equity, and inclusion programs.
Tech Stack
Categories
