AI Engineer, Product
Mistral AI2 months ago
Paris, FranceMid Level / Senior
Responsibilities
- Design and run evaluations for search, chat, document understanding, or audio product areas.
- Define and track metrics covering task success, helpfulness, hallucination proxies, safety, latency, and cost.
- Design, test, and iterate on prompts and system prompts and improve orchestration.
- Run A/B tests on prompts, models, and configurations and make rollout or rollback decisions from data.
- Set up observability for LLM calls using logging, tracing, dashboards, and alerts.
- Operate model releases with canary and shadow traffic, sign-offs, rollback criteria, and regression detection.
- Improve product behaviors such as memory policies, intent classification, routing, tool-call reliability, and retrieval quality.
- Create evaluation templates and documentation for other teams.
- Partner with Science to diagnose regressions and lead post-mortems.
Requirements
- 3–4 years of experience, including production AI/ML experience for software engineers or an ML engineering background moving toward product.
- Strong TypeScript or Python skills.
- Production LLM experience with prompts, tool or function calling, and system prompts.
- Hands-on experience designing evaluations, metrics, and A/B tests.
- Ability to implement directly in product code rather than only in notebooks.
- Experience with observability practices including logging, tracing, dashboards, and alerting.
- Product-oriented approach to forming hypotheses, running experiments, interpreting results, and shipping improvements.
- Preferred experience with safety systems such as moderation, PII handling or redaction, and guardrails.
- Preferred experience with canary or shadow releases, automated rollbacks, or experiment platforms.
- Preferred prior work on search ranking, chat systems, document AI, or audio ML features.
Benefits
- Benefits may include healthcare coverage, parental leave, retirement plans, relocation support, wellness programs, meal allowances, transportation allowances, and other location-specific perks.
- Benefits vary by country and location.
Tech Stack
Categories
About Mistral AI
Mistral AI builds foundation language models and full‑stack AI solutions for enterprises, offering APIs, on‑prem deployment, developer tools, and applications. The privately held company, founded in 2023 and headquartered in Paris, partners with organizations in finance, manufacturing, defense, healthcare, and the public sector to co‑create customized systems. It releases open‑source models alongside commercial offerings and makes its models available through major cloud marketplaces.