Automation Test Engineer
Sonata Software15 hours ago
Bengaluru, IndiaSenior
Responsibilities
- Design and execute test strategies for LLM responses, focusing on accuracy, contextual relevance, long-term context retention, logical reasoning, and hallucination reduction.
- Test autonomous AI agents, including their logic, state management, tool-calling capabilities, and edge-case behavior.
- Develop, maintain, and scale custom automated testing frameworks using Python.
- Integrate AI testing and evaluation frameworks into CI/CD pipelines for continuous model evaluation.
- Define evaluation metrics and heuristics for non-deterministic AI features, including semantic similarity and LLM-as-a-judge approaches.
- Collaborate with AI developers, data scientists, and product managers to establish quality benchmarks for AI-driven features.
Requirements
- BE, B.Tech, or another engineering degree is required.
- Strong proficiency in Python for clean, scalable, and maintainable automation scripts.
- Deep understanding of LLM architectures, prompt engineering, and model behavior.
- Hands-on experience building or testing RAG pipelines and agentic frameworks such as LangChain and LlamaIndex.
- Experience validating non-deterministic outputs using semantic similarity, LLM-as-a-judge methods, or custom evaluation heuristics.
- Understanding of CI/CD tools such as GitHub Actions, GitLab CI, and Jenkins, along with Git version control.
- Experience with AI evaluations, guardrails, and testing or automating AI agents.
- Strong analytical and problem-solving skills, including the ability to anticipate edge cases in generative AI applications.