29 days ago
Gurgaon, IndiaMid Level
Responsibilities
- Design and implement testing strategies for conversational AI, agentic systems, and LLM-powered workflows.
- Develop automated prompt test suites and regression tests for model behavior.
- Create evaluation frameworks, quality metrics, test datasets, and golden examples for GenAI systems.
- Implement monitoring and alerting for production quality degradation.
- Perform adversarial testing for failures, hallucinations, bias, and security vulnerabilities.
- Define acceptance criteria and quality gates with engineering teams.
- Build testing tools and frameworks for GenAI implementations.
- Conduct user acceptance testing and gather internal-user feedback.
- Document testing procedures, known issues, and quality metrics.
- Collaborate with Product, Design, engineering, and data science teams on AI feature quality.
Requirements
- 3+ years of QA or test engineering experience, preferably with AI/ML systems.
- Strong understanding of GenAI, LLMs, prompt engineering, and AI application patterns.
- Experience with test automation and scripting in Python or JavaScript.
- Experience with UI testing using Playwright or Puppeteer.
- Experience with performance testing, scenario testing, and GenAI systems or product development.
- Knowledge of functional, integration, regression, performance, and security testing methodologies.
- Ability to design test cases and evaluation criteria for non-deterministic systems.
- Experience with API testing tools such as Postman and REST Assured, plus backend testing.
- Familiarity with CI/CD pipelines and automated testing integration.
- Preferred experience includes conversational AI, chatbots, agentic systems, ML model evaluation, LLM evaluation frameworks, AI API load testing, responsible AI, enterprise SaaS QA, and test management tools.
- Preferred scripting experience includes Python and working with LLM APIs.
Benefits
- Opportunity to define QA practices for GenAI applications and influence quality standards for enterprise users.
- Collaboration with engineering, data science, Product, and Design teams across the AI product lifecycle.
- Opportunity to develop specialized expertise in GenAI reliability, evaluation, and responsible AI.
Tech Stack
Categories
About Anaplan
Anaplan builds a cloud platform for enterprise planning, forecasting, and scenario modeling used by finance, supply chain, sales, and workforce teams. It sells subscription software and related services to large organizations that need connected, cross-functional planning. Founded in 2006 and headquartered in Miami, Florida, Anaplan was acquired by Thoma Bravo in 2022 after previously trading on the NYSE.