12 hours ago
Pune, IndiaSenior
Responsibilities
- Design, run, and maintain end-to-end business scenarios for multi-step agentic workflows, including routing, tool calls, handoffs, and multi-turn conversations.
- Evaluate agent outputs for correctness, relevance, groundedness, consistency, tone, and completeness through human and automated review.
- Build and maintain evaluation datasets, golden sets, edge cases, adversarial prompts, and automated regression suites.
- Identify cross-agent, orchestration, integration, tool-call, context-loss, looping, and runaway-behavior issues.
- Test for hallucinations, prompt injection, jailbreaks, data leakage, unsafe responses, and out-of-scope behavior.
- Detect and quantify regressions caused by model upgrades, prompt changes, and retrieval or knowledge changes.
- Triage and root-cause issues with Engineering and Product across prompt, model, retrieval, tool, and orchestration layers.
- Re-test fixes, validate end-user experience, maintain current evaluations, and document test strategy, results, and quality trends.
Requirements
- Bachelor’s degree in software engineering, computer science, or a related field.
- 3–5 years of QA experience, including at least 1–2 years testing LLM-based applications, AI agents, or agentic workflows.
- Hands-on experience evaluating non-deterministic outputs and defining pass/fail criteria, rubrics, and acceptance thresholds.
- Experience building or maintaining AI evaluation or regression suites, including approaches such as LLM-as-judge, golden datasets, and scoring rubrics.
- Understanding of tool or function calling, multi-agent orchestration, planning, memory or context, and RAG.
- Experience with test automation tools such as Selenium, Playwright, or TestComplete and with API testing.
Benefits
- Health and well-being programs supporting work-life balance.
- Collaborative culture with growth and professional development opportunities.
- Virtual-first work arrangement with primary work from home and occasional travel to a physical office.
- Compensation packages based on experience and desired skill set.
Tech Stack
Selenium
Categories
About QAD, Inc.
QAD builds cloud-based ERP and supply chain software for manufacturing companies, with offerings like QAD Adaptive ERP and the Redzone Connected Workforce platform, sold primarily by subscription with professional services. Its products target mid-market firms across automotive, industrial, food & beverage, life sciences, and consumer goods, embedding workflow and shop-floor capabilities. Founded in 1979 and headquartered in Miami, QAD is privately held and owned by Thoma Bravo.