AI Evaluation Engineer (QA)
Appnovation TechnologiesResponsibilities
- Run large-scale evaluations across question sets, from human UAT batches to hundreds of thousands or millions of automated evaluations.
- Statistically measure factual grounding and before-and-after accuracy improvements.
- Build a metrics framework demonstrating response-quality improvement over time.
- Design load and quality tests as the content corpus scales.
- Define and maintain test plans, test cases, and quality gates.
- Automate regression and evaluation suites and integrate them into CI/CD pipelines.
- Communicate quality metrics to technical and non-technical stakeholders.
- Collaborate with engineering to reproduce, triage, and verify fixes.
- Continuously improve QA processes and test coverage.
Requirements
- Bachelor’s degree in a technical field or equivalent experience.
- At least 4 years of QA or test-engineering experience with exposure to data and machine-learning systems.
- Strong Python skills and data-science techniques for measuring factual grounding and answer quality.
- Experience with LLM evaluation frameworks and statistical analysis.
- Ability to build large-scale automated evaluation harnesses and lighter human-in-the-loop A/B tests.
- Comfort working with outputs from multiple LLM providers.
- Experience with test-automation frameworks and scripting.
- Strong analytical, communication, problem-solving, decision-making, and collaboration skills.
- Prior consulting experience.
- Life Sciences industry experience and connections are preferred.
Benefits
- Equal Opportunity Employer committed to diversity, equity, and inclusion.
- Accommodations are available upon request throughout the recruitment process.
About Appnovation Technologies
Appnovation is a Global, Full-Service Digital Partner. We design, develop, and deliver digital experiences that drive measurable business transformation. With nearly 20 years of experience and thousands of successful projects delivered, we partner with industry leaders like Pfizer, Google, Yeti, TCL, and AstraZeneca across diverse industries and technologies. Our services combine strategy, user experience, engineering, data, and AI to create purposeful, human-centered solutions. With global delivery and a strong nearshore presence, we offer flexible, cost-effective partnerships tailored to our clients' needs. We build digital foundations through close collaboration, delivering immediate impact and supporting future growth and innovation. Our approach continually evolves to support clients at every stage of their digital journey. We elevate customer experiences, champion digital innovation, and drive scalable business value. Appnovation. Fuelled by experience. Powered by AI. Learn more at appnovation.com. #InspiringPossibility