Research Engineer - Tokenizer & Protocols
Mistral AI3 days ago
Paris, FranceSenior
Responsibilities
- Design, build, and maintain core libraries for tokenization and model interfaces.
- Evolve formats and tokenizers as models and use cases change.
- Develop robust shared tooling from research needs.
- Work across pretraining, post-training, and deployment so training, evaluation, and production systems use consistent formats and tokenizers.
Requirements
- Strong software engineering background with excellent Python skills.
- Excellent understanding of BPE and other tokenization algorithms, special tokens, chat templates, and tokenizer training.
- Hands-on experience with tiktoken or Hugging Face tokenizers is ideal.
- Experience building production-grade libraries used by many people.
- Good knowledge of how LLMs are trained and served.
- Clear communication, attention to detail, and a collaborative mindset.
- Open-source maintenance experience is a plus.
- Experience with inference engines is a plus.
Benefits
- Comprehensive benefits may include healthcare coverage, parental leave, retirement plans, relocation support, wellness programs, and meal and transportation allowances, varying by country.
- Primarily based in the Paris or London offices; remote candidates may be considered in France or the UK.
- Remote hires must visit the local office during their first onboarding week, with accommodation and travel covered, and then at least four days per month.
About Mistral AI
Mistral AI builds foundation language models and full‑stack AI solutions for enterprises, offering APIs, on‑prem deployment, developer tools, and applications. The privately held company, founded in 2023 and headquartered in Paris, partners with organizations in finance, manufacturing, defense, healthcare, and the public sector to co‑create customized systems. It releases open‑source models alongside commercial offerings and makes its models available through major cloud marketplaces.