Test Automation Lead – AI Testing
Eli Lilly and Company24 days ago
Hyderābād, IndiaStaff+
Responsibilities
- Lead enterprise-wide test automation and quality engineering strategy for AI-powered systems and automation solutions.
- Design, develop, maintain, and execute automated tests covering UI, APIs, data, model outputs, regression, performance, integration, and AI/ML workflows.
- Build automation frameworks for model training, inference, integration, synthetic data, explainability validation, and continuous model monitoring.
- Define and track AI quality metrics including accuracy, precision, recall, and fairness, and integrate them into CI/CD pipelines.
- Apply AI evaluation, model benchmarking, prompt validation, prompt injection testing, adversarial testing, bias detection, and explainability testing.
- Lead and mentor global and cross-functional QA teams and collaborate with data scientists, ML engineers, developers, product owners, architects, and senior leadership.
- Own test strategies, risk mitigation, QA deliverables, process improvements, and continuous testing initiatives across enterprise projects.
- Provide expert consultation on AI and automation tools, frameworks, and scalable solutions while evaluating emerging technologies.
Requirements
- Bachelor’s or master’s degree in computer science, data science, information technology, or a related field.
- At least 7 years of test automation experience, including at least 2 years focused on AI/ML testing.
- Proven experience designing and executing automated test suites for complex, distributed, or AI-based systems.
- Strong programming and problem-solving skills using Python, JavaScript, TypeScript, or similar languages.
- Deep expertise in test automation frameworks and strategies, including testing AI/ML models, data pipelines, and intelligent applications.
- Hands-on experience with AI evaluation methodologies, model benchmarking, quality metrics, LLMs, generative AI, RAG, and agentic AI architectures.
- Experience with RAGAS, DeepEval, LLM-as-a-Judge, Langfuse, Playwright, Cypress, Postman, k6, and JMeter.
- Knowledge of supervised and unsupervised learning, ML platforms, cloud AI services, MLOps, automated model lifecycle management, data quality, data drift, and training/inference pipeline testing.
- Knowledge of Jenkins, GitHub Actions, Azure DevOps, and CI/CD practices.
- Preferred experience with DeepChecks, Great Expectations, DataFold, Power BI, Splunk, or comparable monitoring and reporting tools.
- Preferred certification in AI/ML, test automation, or cloud technologies, such as AWS Certified Machine Learning or ISTQB Advanced Test Automation Engineer.
- Strong analytical, debugging, communication, stakeholder management, leadership, and mentoring skills.
Benefits
- Travel is expected at 0–10%.
- The position does not support sponsorship.
- The role offers enterprise-wide scope, cross-functional collaboration, and opportunities to lead AI quality engineering innovation.
Tech Stack
Categories