Test Automation Lead – AI Testing
Eli Lilly and Company2 months ago
Hyderābād, IndiaStaff+
Responsibilities
- Lead enterprise-wide test automation and quality engineering strategy for AI-powered systems and automation solutions.
- Design, develop, maintain, and execute automated tests covering UI, APIs, data, model outputs, regression, performance, integration, and AI/ML workflows.
- Build automation frameworks for model training, inference, integration, synthetic data, explainability validation, and continuous model monitoring.
- Define and track AI quality metrics including accuracy, precision, recall, and fairness, and integrate them into CI/CD pipelines.
- Apply AI evaluation, model benchmarking, prompt validation, prompt injection testing, adversarial testing, bias detection, and explainability testing.
- Lead and mentor global and cross-functional QA teams and collaborate with data scientists, ML engineers, developers, product owners, architects, and senior leadership.
- Own test strategies, risk mitigation, QA deliverables, process improvements, and continuous testing initiatives across enterprise projects.
- Provide expert consultation on AI and automation tools, frameworks, and scalable solutions while evaluating emerging technologies.
Requirements
- Bachelor’s or master’s degree in computer science, data science, information technology, or a related field.
- At least 7 years of test automation experience, including at least 2 years focused on AI/ML testing.
- Proven experience designing and executing automated test suites for complex, distributed, or AI-based systems.
- Strong programming and problem-solving skills using Python, JavaScript, TypeScript, or similar languages.
- Deep expertise in test automation frameworks and strategies, including testing AI/ML models, data pipelines, and intelligent applications.
- Hands-on experience with AI evaluation methodologies, model benchmarking, quality metrics, LLMs, generative AI, RAG, and agentic AI architectures.
- Experience with RAGAS, DeepEval, LLM-as-a-Judge, Langfuse, Playwright, Cypress, Postman, k6, and JMeter.
- Knowledge of supervised and unsupervised learning, ML platforms, cloud AI services, MLOps, automated model lifecycle management, data quality, data drift, and training/inference pipeline testing.
- Knowledge of Jenkins, GitHub Actions, Azure DevOps, and CI/CD practices.
- Preferred experience with DeepChecks, Great Expectations, DataFold, Power BI, Splunk, or comparable monitoring and reporting tools.
- Preferred certification in AI/ML, test automation, or cloud technologies, such as AWS Certified Machine Learning or ISTQB Advanced Test Automation Engineer.
- Strong analytical, debugging, communication, stakeholder management, leadership, and mentoring skills.
Benefits
- Travel is expected at 0–10%.
- The position does not support sponsorship.
- The role offers enterprise-wide scope, cross-functional collaboration, and opportunities to lead AI quality engineering innovation.
Tech Stack
Categories
About Eli Lilly and Company
Eli Lilly and Company researches, develops, manufactures, and markets prescription medicines and biologics for patients, sold through healthcare providers, pharmacies, and payers. Core therapeutic areas include diabetes, oncology, immunology, neuroscience, and obesity. Founded in 1876 and headquartered in Indianapolis, it is a public company (NYSE: LLY) with global operations and is building a new advanced manufacturing site for gene therapy and other modalities in Lebanon, Indiana.