17 hours ago
Bengaluru, IndiaSenior
Responsibilities
- Design evaluation strategies for agent behavior, task completion, retrieval quality, groundedness, safety, and reliability.
- Build automated evaluation harnesses, regression suites, benchmark datasets, and production quality gates.
- Evaluate multi-step agent trajectories, tool use, planning, recovery, and human escalation.
- Combine deterministic tests, model-based evaluation, human review, and production telemetry.
- Perform failure analysis, red teaming, and root-cause investigation.
- Integrate evaluations into CI/CD, release, monitoring, and incident-management processes.
- Define scorecards for engineering, risk, product, and business stakeholders.
- Architect and build custom AI infrastructure and hardware solutions, and optimize performance, power consumption, cost, and scalability.
- Advise on AI infrastructure technology and vendor evaluation, selection, and full-stack integration.
Requirements
- At least 5 years of professional experience.
- 15 years of full-time education.
- Large Language Model experience is required.
- Experience building evaluation or quality systems for production AI, ML, search, or decision systems is required.
- Expertise with LangSmith, Braintrust, Arize Phoenix, Weights & Biases Weave, MLflow, and OpenAI Evals.
- Proficiency in Python, test automation, data analysis, statistics, and experimentation.
- Experience with LLM and RAG evaluation, agent trajectory analysis, benchmark design, and error taxonomies.
- Knowledge of tracing, observability, adversarial testing, safety testing, and production monitoring.
- Ability to distinguish model, retrieval, prompt, tool, data, and orchestration failures.
- Experience using Codex, Claude Code, or Cursor to support evaluation and quality engineering work.
Tech Stack
MLflowPython
Categories
About Accenture
Accenture is a global professional services firm providing management consulting, systems integration and technology, cybersecurity, and business process outsourcing for enterprises and governments. It operates a services-driven model delivering projects and managed services, often with major cloud and software partners, across industries worldwide. Headquartered in Dublin and publicly traded on the NYSE (ACN), it originated as Andersen Consulting and adopted the Accenture name in 2001.