AI Quality Engineer
Aviva Investors21 hours ago
Warsaw, PolandMid Level
Responsibilities
- Translate business expectations, user outcomes, and operational risks into measurable AI evaluation criteria.
- Define successful, partially successful, and unsuccessful system behavior across conversational, multi-turn, and non-deterministic AI systems.
- Build and maintain evaluation datasets covering common, rare, boundary, and high-risk scenarios.
- Assess dataset coverage, class balance, duplicates, data leakage, label quality, synthetic-versus-production differences, and freshness.
- Select manual, deterministic, model-based, and statistical evaluation methods.
- Analyze errors by category, user journey, risk level, and likely root cause.
- Distinguish failures in models, retrieval, chunking, ranking, prompts, tools, guardrails, integrations, source data, and evaluators.
- Validate automated evaluators, including LLM-as-a-Judge, against human judgment.
- Detect blind spots, noise, bias, and misleading aggregation in evaluation results.
- Create baselines, controlled comparisons, and regression tests for system versions.
- Prioritize testing based on the probability and impact of failure.
- Communicate results, uncertainty, limitations, and residual risks to support release and product decisions.
- Identify emerging failure modes based on system changes, production observations, and research.
Requirements
- Professional experience in quality engineering, software testing, test automation, data science, software engineering, or a related field; different seniority levels may be considered.
- Experience or strong interest in developing, testing, evaluating, or supporting AI-powered solutions.
- Good Python proficiency and experience writing clean, maintainable code for analysis, automation, or testing.
- Strong analytical, problem-solving, root-cause investigation, and communication skills.
- Ability to translate business objectives, customer outcomes, and risks into practical evaluation approaches.
- Interest in AI system behavior, reliability measurement, emerging technologies, and system robustness.
- Preferred: experience evaluating, testing, or monitoring AI-powered systems in production.
- Preferred: experience with structured data, JSON, logs, and tabular evaluation results.
- Preferred: understanding of LLM behavior, tokenization, context limits, parameters, non-determinism, multi-turn context, RAG, grounding, faithfulness, hallucinations, guardrails, prompt injection, and tool or function calling.
- Preferred: familiarity with Git, modular code, configuration management, and automated tests.
Benefits
- Contract of employment with a performance bonus.
- Private medical care through ENEL-MED, cafeteria benefits including a MultiSport card, employer-funded group insurance, and an Employee Shares Plan.
- Paid time off, an extra day off for birthdays, paid volunteer leave, additional tenure-based leave, and two weeks of paid Wellbeing Leave after 10 years of service.
- Parental benefits including six weeks of leave for a second parent and shortened working time after parental leave.
- Psychological support consultations, e-learning platforms, and an internal employee referral program.
- Hybrid work from Poland with a few days per month in the Warsaw office.
- Additional entitlements for people with disabilities and an inclusive, diversity-focused work environment.
About Aviva Investors
Aviva Investors is the global asset management business of Aviva plc, headquartered in London and founded in 2008. It manages fixed income, equity, real assets, multi-asset, and alternative strategies, including ESG-oriented mandates, for institutional, wholesale, and retirement clients worldwide. Its business model centers on management and performance fees from pooled funds and segregated mandates across global markets.