2 days ago
Pune, IndiaMid Level / Senior
Responsibilities
- Design and maintain end-to-end business scenarios covering agent routing, tool and function calls, orchestration, handoffs, and multi-turn conversations.
- Evaluate agent responses for correctness, relevance, groundedness, consistency, tone, and completeness using human and automated methods.
- Build and maintain evaluation datasets, golden sets, edge cases, adversarial prompts, and automated regression suites.
- Identify cross-agent, orchestration, context, integration, tool-call, looping, and runaway-behavior issues.
- Test for hallucinations, prompt injection, jailbreaks, data leakage, and unsafe or out-of-scope responses.
- Detect and quantify regressions caused by model, prompt, retrieval, or knowledge changes.
- Triage issues with Engineering and Product, determine root causes, re-test fixes, and validate the end-user experience.
- Document test strategy, evaluation results, and quality trends for global stakeholders.
Requirements
- Bachelor's degree in software engineering, computer science, or a related field.
- 3-5 years of QA experience, including at least 1-2 years testing LLM-based applications, AI agents, or agentic workflows.
- Hands-on experience evaluating non-deterministic outputs by defining pass/fail criteria, rubrics, and acceptance thresholds.
- Experience building or maintaining AI evaluation or regression suites, including LLM-as-judge methods, golden datasets, or scoring rubrics.
- Understanding of tool and function calling, multi-agent orchestration, planning, memory and context, and RAG.
- Experience with test automation tools such as Selenium, Playwright, or TestComplete, and with API testing.
Benefits
- Health and well-being programs supporting work-life balance.
- Collaborative culture with growth and advancement opportunities.
- Virtual-first work arrangement with primary work performed from home.
- Occasional travel to a physical office may be required for collaboration, design, strategy, and alignment.
- Compensation packages are based on experience and desired skill set.
Tech Stack
Selenium
Categories
About QAD, Inc.
QAD builds cloud-based ERP and supply chain software for manufacturing companies, with offerings like QAD Adaptive ERP and the Redzone Connected Workforce platform, sold primarily by subscription with professional services. Its products target mid-market firms across automotive, industrial, food & beverage, life sciences, and consumer goods, embedding workflow and shop-floor capabilities. Founded in 1979 and headquartered in Miami, QAD is privately held and owned by Thoma Bravo.