Evaluation Engineer, Applied AI
Mistral AI2 days ago
Paris, FranceMid Level
Responsibilities
- Design and implement comprehensive frameworks for evaluating LLM performance across customer use cases.
- Build and maintain scalable evaluation infrastructure and pipelines for rapid, reproducible assessment.
- Develop evaluation methods for sector-specific and emerging model capabilities.
- Collaborate with customers to create custom evaluation suites and criteria.
- Partner with research teams to translate evaluation insights into model improvements.
- Work with product teams to refine evaluation tooling based on user feedback.
- Define and communicate success metrics for production-ready AI models.
Requirements
- At least 3 years of experience evaluating ML, LLM, or agentic systems.
- Experience implementing AI or machine learning products, especially through APIs or back-end systems.
- Strong Python coding skills.
- Deep understanding of machine learning concepts and algorithms as they relate to LLMs.
- Experience communicating technical topics to technical and non-technical audiences.
- Familiarity with LM Eval Harness, OpenAI Evals, or related open-source evaluation projects.
- Experience with ML libraries such as PyTorch and HuggingFace Transformers.
- Collaborative communication style and an outcomes-focused, low-ego approach.
Benefits
- Benefits may include healthcare coverage, parental leave, retirement plans, relocation support, wellness programs, meal allowances, and transportation allowances, varying by country.
- Benefits and practical support are location-specific, with details provided on the company's Benefits page.
Categories
About Mistral AI
Mistral AI builds foundation language models and full‑stack AI solutions for enterprises, offering APIs, on‑prem deployment, developer tools, and applications. The privately held company, founded in 2023 and headquartered in Paris, partners with organizations in finance, manufacturing, defense, healthcare, and the public sector to co‑create customized systems. It releases open‑source models alongside commercial offerings and makes its models available through major cloud marketplaces.