AI Test Engineer
Ather Energy5 days ago
Bengaluru, IndiaEntry Level / Mid Level
Responsibilities
- Design, develop, and execute automated test plans and validation scripts for AI applications, LLMs, and autonomous agents.
- Create testing strategies for model accuracy, response relevance, hallucination detection, and prompt injection risks.
- Validate multi-agent workflows, task routing, state transitions, and pipeline error handling.
- Build benchmarking frameworks for model accuracy, latency, and performance under stress and load.
- Create mock environments, test datasets, and simulation frameworks for edge cases and complex agent behavior.
- Perform continuous regression testing as AI models are updated or fine-tuned.
- Identify, document, and isolate complex bugs while collaborating with Data Science, Development, and Product teams.
- Create real-time automated reporting dashboards for model quality and system reliability metrics.
Requirements
- Bachelor’s degree in Computer Science, Software Engineering, or an equivalent technical discipline.
- 1–3 years of experience in software engineering, testing, or test automation.
- At least 6 months to 1 year of experience validating AI agents, LLM systems, or conversational agents.
- Experience designing and deploying AI/LLM evaluation frameworks such as Ragas or Deepeval.
- Advanced Python test automation using PyTest, Selenium, or Playwright.
- Understanding of LLM parameters, prompt engineering, agent state machines, and JSON API payloads.
- Knowledge of software testing methodologies, regression automation, and bug lifecycle management using JIRA.
- Preferred experience with GitLab CI, GitHub Actions, Docker, Kubernetes, GCP, Grafana, Qdrant, Milvus, Langfuse, and MLflow.
- Experience testing web or mobile applications with integrated AI or conversational interfaces is preferred.