EY - GDS Consulting - AIA - Gen AI - Senior
Ernst and Young25 days ago
Pune, IndiaSenior
Responsibilities
- Design, develop, and execute comprehensive testing strategies for AI, Generative AI, Agentic AI, and Agentic RAG systems.
- Validate LLM outputs for accuracy, consistency, relevance, safety, compliance, bias, hallucinations, toxicity, and robustness.
- Develop Python-based automated test frameworks and validation utilities for AI workflows, APIs, models, and orchestration pipelines.
- Create evaluation frameworks and dashboards for RAG retrieval quality, answer correctness, faithfulness, relevance, grounding, and response effectiveness using Ragas and similar tools.
- Perform functional, integration, regression, performance, reliability, scalability, API, adversarial, penetration, prompt injection, and jailbreak testing.
- Validate AI integrations across enterprise applications, APIs, databases, cloud platforms, retrieval pipelines, vector databases, embeddings, and enterprise search solutions.
- Collaborate with AI engineers, architects, developers, data scientists, product teams, and business stakeholders on risk identification, design reviews, model evaluations, governance, and production readiness.
- Develop reusable testing assets, quality standards, governance controls, metrics, defect-reporting processes, automation frameworks, and best practices.
- Support presales activities, technical demonstrations, innovation initiatives, client workshops, and mentorship of junior QA engineers.
Requirements
- 5–9+ years of total professional experience, including at least 3 years of directly relevant hands-on experience in AI/ML, applied AI, or Generative AI systems.
- Bachelor’s, master’s, or PhD in computer science, data science, engineering, or a related field.
- Professional implementation experience with at least two delivered AI implementations; personal projects, cloned demos, certifications, and tutorials alone are insufficient.
- Strong hands-on depth in at least three relevant areas, with meaningful ownership in at least two, including LLM applications, retrieval pipelines, agent orchestration, evaluation or guardrails, model adaptation, cloud deployment, conversational AI, multimodal AI, or backend engineering for AI systems.
- Strong software engineering, quality engineering, test automation, secure development, troubleshooting, analytical, root-cause analysis, communication, and presentation skills.
- Experience with AI testing strategies, validation frameworks, automated test suites, RAG evaluation, Agentic AI workflows, prompt engineering validation, reasoning evaluation, observability, monitoring, and performance benchmarking.
- Experience with Ragas or similar evaluation frameworks and with testing AI integrations across APIs, enterprise applications, databases, and cloud services.
- Experience with Microsoft Azure AI Platform, Azure AI services, Python, FastAPI-based services, and automated API testing frameworks.
- Knowledge of Responsible AI, AI safety controls, governance, regulatory compliance, vulnerability assessment, retrieval augmentation architectures, vector databases, embeddings, and enterprise search.
- Experience with functional, regression, integration, end-to-end, performance, scalability, reliability, security, adversarial, prompt injection, jailbreak, and production-readiness testing.
Benefits
- Work on AI-driven innovation projects for global clients across multiple sectors.
- Collaborate with AI experts, analytics leaders, and industry specialists in a global, entrepreneurial environment.
- Access continuous learning, coaching, leadership development, and career-building resources.
- Work within EY Global Delivery Services across Argentina, China, India, the Philippines, Poland, and the UK.
- Participate in a diverse and inclusive culture with flexibility and support to make a meaningful impact.
Tech Stack
FastAPIPython
Categories
About Ernst and Young
Ernst & Young (EY) provides audit/assurance, tax, consulting, strategy and transactions services to enterprises, financial institutions, and public‑sector clients. Structured as a global network of partner‑owned member firms, it sells professional services on a fee basis, including a dedicated Financial Services Organization for banking, insurance, and capital markets. Headquartered in London, EY was formed in 1989 from the merger of Ernst & Whinney and Arthur Young, and operates in 150+ countries.