1 day ago
Amsterdam, NetherlandsMid Level
Responsibilities
- Automate quality and performance evaluations for multi-turn conversations, long-horizon tasks, autonomous agents, and multi-step workflows.
- Implement schema validation, tracing, calibrated LLM-as-a-Judge frameworks, filters, golden datasets, and human-in-the-loop reviews.
- Conduct red-team, prompt-injection, jailbreak, privacy, PII-leak, threat-modeling, and model-safety testing.
- Embed evaluation test harnesses in CI/CD and monitor production telemetry for regressions, behavioral drift, and recovery.
- Maintain evaluation data lineage, versioning, curation, and golden responses.
- Translate business risks into measurable quality criteria, controls, release recommendations, and go/no-go decisions.
- Align stakeholders on journeys, edge cases, thresholds, mitigations, testability, observability, governance, and evaluation strategy.
- Communicate residual risks, coach teams, and share reusable quality assets and standards.
Requirements
- Bachelor's or master's degree in Computer Science, Data Science, or a related quantitative field.
- At least 3 years of experience in machine learning QA, AI evaluation engineering, or MLOps engineering.
- Dutch proficiency at B2+ level and English proficiency at C2 level.
- Experience evaluating autonomous agents, multi-step workflows, or Retrieval-Augmented Generation applications.
- Strong Python proficiency and experience with MLflow, Galileo, and PyRIT.
- Deep knowledge of prompt engineering, vector databases, tool schema design, and execution telemetry.
- Hands-on experience with adversarial testing, threat modeling, or model safety guardrails.
- Strong communication, technical leadership, growth mindset, and commitment to continuous innovation.
- TMAP or ISTQB certification and open-source or engineering-community contributions are advantageous.
Benefits
- 4x9 workweek option, hybrid working, and FlexDay.
- Competitive base salary, MyBenefits Budget, annual bonus, net allowance, and employee share purchase plan.
- Collective health insurance, mental health support, and a well-being hub.
- 26 vacation days, special leave types, and Culture Days.
- NS Business Card and mobility budget options including an electric bike, car lease, or public transport reimbursement.
- Performance Achievement Program, learning and development platform, and guidance from People Leads.
- Do Good Volunteer Program, inclusion and diversity networks, Refugee Talent Track, company events, and a flat organizational hierarchy.
About Accenture
Accenture is a global professional services firm providing management consulting, systems integration and technology, cybersecurity, and business process outsourcing for enterprises and governments. It operates a services-driven model delivering projects and managed services, often with major cloud and software partners, across industries worldwide. Headquartered in Dublin and publicly traded on the NYSE (ACN), it originated as Andersen Consulting and adopted the Accenture name in 2001.