Agentic AI Evaluation Engineer
Ernst and Young2 months ago
Kolkāta, IndiaSenior
Responsibilities
- Define and operationalize evaluation strategies for GenAI, RAG, and agentic AI use cases.
- Translate business use cases into evaluation plans covering datasets, metrics, thresholds, red-team scenarios, and reporting requirements.
- Create reusable evaluation templates, test libraries, scoring rubrics, dataset requirements, and statistical coverage guidance.
- Build Python-based evaluation harnesses and pipelines for answer quality, grounding, agent behavior, operational quality, and safety.
- Combine LLM-as-judge and human evaluation using calibrated rubrics, sampling plans, and agreement checks.
- Execute red-team testing for prompt injection, jailbreaks, sensitive-data disclosure, insecure output handling, tool hijacking, retrieval poisoning, and model denial-of-service or denial-of-wallet patterns.
- Integrate evaluations into development workflows through regression gates, CI checks, and benchmark comparisons across models, prompts, tools, and retrievers.
- Recommend mitigations including input validation, retrieval filtering, tool sandboxing, least-privilege permissions, guardrails, refusal logic, output encoding, and monitoring alerts.
- Produce auditable evaluation reports with methodology, datasets, metrics, thresholds, results, risk assessments, and recommended control actions.
- Present residual risks, limitations, and go/no-go recommendations to product, architecture, risk, compliance, and assurance stakeholders.
- Coordinate multiple projects and initiatives and apply project management, prioritization, and reporting practices.
Requirements
- Bachelor’s or master’s degree in data science, statistics, engineering, operational research, or a related field with a strong focus on modern data architectures, processes, and environments.
- 4–7+ years of relevant experience in ML, AI, GenAI, agentic engineering, NLP/LLMs, evaluation engineering, applied research, security testing, red teaming, or evaluation-harness development.
- Strong hands-on Python experience for data processing, metric computation, orchestration, and reporting pipelines.
- Practical understanding of RAG, embeddings and vector search, prompt orchestration, tool calling, multi-agent systems, memory, and routing.
- Experience designing evaluation metrics, rubrics, automated scoring, sampling strategies, and regression methods.
- Understanding of GenAI risks and mitigations including data leakage, hallucinations, prompt injection, unsafe content, and bias.
- Understanding of OWASP Top 10 for LLM Applications and experience translating risks into test cases and controls.
- Experience with adversarial testing such as jailbreak prompts, injection patterns, tool misuse, and retrieval poisoning.
- Familiarity with secure-by-design practices for LLM applications, including least privilege, safe tool invocation, output validation, encoding, and monitoring.
- Experience with RAGAS, DeepEval, LangSmith, Phoenix/Arize, or custom evaluation harnesses.
- Experience with A/B testing, baseline comparisons, statistical rigor, structured logging, observability, and tracing.
- Basic DevOps experience with Git, CI/CD, Docker, and reproducible environments.
- Strong written, oral, presentation, facilitation, analytical, organizational, and stakeholder-influence skills.
- Ability to operate in ambiguity, challenge assumptions constructively, prioritize multiple initiatives, and produce pragmatic solutions.
- Preferred qualifications include experience in assurance, finance, regulatory, model validation, risk acceptance, audit evidence, responsible AI frameworks, multilingual evaluation, domain-heavy enterprise assistants, and the Azure ecosystem.
Benefits
- Support, coaching, and feedback from experienced colleagues.
- Opportunities to develop new skills and progress your career.
- Individual progression planning, education, coaching, and practical experience.
- Interdisciplinary work environment with challenging assignments and knowledge exchange.
- Freedom and flexibility to handle the role in a way that is right for you.
- Opportunity to work with EY GDS Assurance practices globally and with businesses across multiple industries.
About Ernst and Young
Ernst & Young (EY) provides audit/assurance, tax, consulting, strategy and transactions services to enterprises, financial institutions, and public‑sector clients. Structured as a global network of partner‑owned member firms, it sells professional services on a fee basis, including a dedicated Financial Services Organization for banking, insurance, and capital markets. Headquartered in London, EY was formed in 1989 from the merger of Ernst & Whinney and Arthur Young, and operates in 150+ countries.