Senior AI Quality/Evaluation Engineer
International Data Group, Inc.3 days ago
Prague, CzechiaSenior
Responsibilities
- Design and build evaluation infrastructure for accurate, well-sourced, high-quality AI responses.
- Build automated test suites and regression detection systems for agent pipeline changes.
- Create evaluation frameworks for response accuracy, citation correctness, groundedness, relevance, and source quality.
- Translate product quality criteria into measurable, automatable test scenarios.
- Build cost and latency monitoring for AI pipeline execution.
- Define scalable evaluation standards and practices while independently establishing the function’s roadmap.
Requirements
- 6+ years of software engineering experience with significant work in testing infrastructure, ML evaluation, or quality systems.
- Experience building evaluation or testing frameworks for LLM-based or ML-based systems.
- Understanding of generative-AI response-quality measures including accuracy, groundedness, citation correctness, and relevance.
- Proficiency in Python.
- Ability to operate independently and define a roadmap as the first hire in the function.
- Experience translating qualitative quality criteria into quantitative measurements at the intersection of engineering and product.
- Preferred: experience with RAGAS, DeepEval, or custom LLM evaluation frameworks.
- Preferred: familiarity with Langfuse, LangSmith, or Weights & Biases.
- Preferred: statistical methods for quality measurement, A/B testing or experimentation infrastructure, and search relevance or information-retrieval metrics.
Benefits
- Five weeks of holidays plus additional corporate days off.
- Full-time, permanent position in a hybrid model based in Prague or Ostrava, with 2–3 office days per week and flexibility to work from home most of the week.
- Sick days and flexibility to schedule working hours.
- Cafeteria system with Flexipasses, pension or life insurance, and Multisport card options.
- Meal allowance.
Tech Stack
Categories
About International Data Group, Inc.
International Data Group, Inc. (IDG) provides technology market intelligence and B2B marketing solutions through two main businesses: IDC (research, data, and advisory) and Foundry (media, events, and marketing technology). Its revenue comes from subscriptions and data licenses, consulting, events, advertising, and SaaS marketing tools. Founded in 1964 and headquartered in Boston, the company is owned by Blackstone and operates well-known tech media brands such as Computerworld and PCWorld.