Quality Assurance Engineer – AI Agentic Applications, AVP
D.B. Group9 hours ago
Pune, IndiaStaff+
Responsibilities
- Design, execute, and maintain test strategies and automation frameworks for AI-powered agentic workflows.
- Develop specialized testing for LLM applications, including response accuracy, relevance, grounding, hallucinations, and edge cases.
- Evaluate autonomous and semi-autonomous agents for planning, tool selection and sequencing, agent interactions, and error or timeout recovery.
- Validate RAG performance, including document ingestion, chunking, metadata quality, embeddings, and citation or source attribution.
- Perform AI safety and security testing for prompt injection, data leakage, unauthorized tool execution, and privilege escalation.
- Establish quantitative evaluation metrics and quality gates for accuracy, groundedness, hallucination rate, tool-selection accuracy, cost, and latency.
- Build and maintain automated API, UI, integration, RAG, and agent-trajectory test suites for continuous evaluation.
- Analyze production AI interactions and failed agent journeys, create regression tests from incidents, and support decision auditability.
Requirements
- Bachelor's degree or above in Computer Science, Engineering, Information Technology, or a related discipline.
- At least 5 years of software quality assurance and testing experience, including test planning, functional, integration, regression, and end-to-end testing.
- 1–2 years of hands-on experience testing GenAI, LLM, or agentic AI applications.
- Strong experience building test automation frameworks and utilities rather than relying solely on manual tools.
- Strong programming ability in at least one of Java, Kotlin, or Python.
- Understanding of LLMs, prompt engineering, RAG, embeddings, vector databases, AI agents, agentic workflows, and tool or function calling.
- Hands-on experience with LLM and agent evaluation tools such as Langsmith, Langfuse, Ragas, or equivalent frameworks.
- Nice-to-have experience with API and data integrations, REST APIs, JSON, SQL, databases, Kafka, CI/CD quality gates, distributed systems, and financial services workflows.
- Nice-to-have experience with LangChain, LangGraph, LlamaIndex, Vertex AI, OpenAI, prompt injection testing, AI evaluation, and AI observability.
- Ability to use AI tools productively while applying critical judgment for responsible and ethical use of data and AI outputs.
Benefits
- Best-in-class leave policy.
- Gender-neutral parental leave.
- 100% reimbursement under the childcare assistance benefit.
- Sponsorship for industry-relevant certifications and education.
- Employee Assistance Program for employees and family members.
- Comprehensive hospitalization insurance for employees and dependents.
- Accident and term life insurance.
- Complementary health screening for employees aged 35 and above.
- Training and development, coaching and expert support, continuous learning, and flexible benefits.
- The role is located in Pune, India.
About D.B. Group
D.B. Group is an Italian freight forwarder and logistics provider offering air, ocean, road, and rail transport, customs brokerage, and supply chain management for global shippers. Founded in 1980 and headquartered in Montebelluna, Veneto, it is privately held and operates through 79 subsidiaries with a worldwide partner network. The company delivers end-to-end forwarding and integrated logistics services, from consolidation and warehousing to door-to-door deliveries and trade compliance.