QA Engineer
Newcode.ai8 days ago
Oslo, NorwayMid Level
Responsibilities
- Design and build automated and human-in-the-loop evaluation frameworks for AI-powered products and workflows.
- Develop metrics and benchmarks for accuracy, reliability, consistency, and overall AI quality.
- Build evaluation datasets, test cases, and regression suites for complex legal use cases.
- Analyze model outputs to identify failure modes, patterns, and improvement opportunities.
- Translate real-world legal workflows into rigorous evaluation criteria with Legal Engineers.
- Integrate evaluations into product and development workflows with Software Engineers.
- Establish automated testing and monitoring to detect quality regressions before release.
- Evaluate changes to models, prompts, retrieval systems, agents, and workflows.
- Develop tooling to make AI quality measurable and visible across the organization.
- Investigate unexpected model behavior, determine root causes, and establish quality standards for legal use cases.
Requirements
- Strong software engineering or technical background with experience working on AI/ML systems.
- Experience building testing, evaluation, or quality frameworks for AI-powered products.
- Strong Python and/or similar programming experience.
- Understanding of LLMs, generative AI, retrieval systems, agents, and/or AI evaluation methodologies.
- Strong analytical, problem-solving, communication, and attention-to-detail skills.
- Ability to turn ambiguous problems into measurable technical frameworks.
- Ability to communicate technical findings to technical and non-technical stakeholders.
- Comfort working cross-functionally with Engineering, Product, and Legal Engineering.
Benefits
- Hybrid workplace in Oslo, Norway.
- Culture of trust, excellence, reliability, respect, initiative, and inclusion.
- Collaborative environment working on AI-powered legal engineering solutions.
- Opportunity for growth, ownership, and shaping the role in a fast-growing company.