AI Quality Engineering Team Lead
Aviva Investors21 hours ago
Warsaw, PolandStaff+
Responsibilities
- Lead quality engineering from AI risk assessment through release and monitoring.
- Evaluate multi-turn, RAG, and agentic AI behavior, including grounding, guardrails, and tool use.
- Build representative datasets and assess coverage, balance, leakage, label quality, and freshness.
- Assess correctness, relevance, hallucination, safety, bias, privacy, and behavioral drift.
- Investigate unexpected outcomes across data, prompts, models, retrieval, integration, orchestration, and evaluators.
- Build reusable evaluation components, automated test pipelines, reporting, and evidence-generation patterns.
- Validate automated evaluators against human judgment and use metrics and risk-based regression tests to support release decisions.
- Integrate evaluation into CI/CD, observability, and engineering workflows with traceability.
- Produce concise, auditable findings covering confidence, uncertainty, customer impact, and residual risk.
- Partner with product, engineering, architecture, data, risk, compliance, and quality stakeholders.
- Coach and guide AI Quality Engineers, plan team work, review outputs, and foster constructive challenge and learning.
Requirements
- Several years of relevant professional experience in quality engineering, software testing, software engineering, data science, or a related technical field.
- Hands-on experience evaluating AI, large language model, RAG, or agent-based applications.
- Practical Python skills and experience with APIs, logs, traces, structured datasets, and SQL or similar analysis.
- Experience designing repeatable evaluations and automation rather than relying only on manual testing or vendor scoring.
- Understanding of sampling, thresholds, trends, and non-deterministic system behavior.
- Experience with automated testing, version control, CI/CD, and maintainable engineering practices.
- Ability to turn technical findings into decision-quality evidence for business, risk, and engineering stakeholders.
- Experience coaching or leading engineers in an emerging or ambiguous domain.
- Strong communication and influencing skills, including the confidence to challenge weak evidence.
- Preferred experience leading projects, coordinating technical workstreams, or managing people.
- Preferred experience with AI evaluation or observability tools such as DeepEval or Arize.
- Preferred experience designing shared frameworks, reusable services, or platform capabilities.
- Knowledge of responsible AI, model risk, controls, privacy, or customer-outcome requirements is advantageous.
- Experience in financial services, insurance, or another regulated industry is advantageous.
- Exposure to cloud AI systems, observability, and production support is advantageous.
Benefits
- Contract of employment and performance bonus.
- Private medical care through ENEL-MED and a cafeteria benefits system such as a MultiSport card.
- Six weeks of baby leave for the second parent and shortened working time after parental leave.
- Employer-funded group insurance and an employee shares plan.
- Extra day off for birthdays, paid time off to volunteer, and additional leave earned over time.
- Two weeks of paid Wellbeing Leave after 10 years of service.
- Psychological support consultations and access to e-learning platforms.
- Internal employee referral program.
- Hybrid working from Poland with a few days per month in the Warsaw office.
- Additional entitlements for people with disabilities and a workplace promoting diversity.
About Aviva Investors
Aviva Investors is the global asset management business of Aviva plc, headquartered in London and founded in 2008. It manages fixed income, equity, real assets, multi-asset, and alternative strategies, including ESG-oriented mandates, for institutional, wholesale, and retirement clients worldwide. Its business model centers on management and performance fees from pooled funds and segregated mandates across global markets.