29 days ago
Toronto, Canada +2 moreMid Level
Responsibilities
- Develop modeling and experimentation strategies for voice, language, reasoning, ordering, multilingual, and multimodal agent problems.
- Build offline and online evaluations measuring task completion, accuracy, safety, latency, cost, conversational quality, and business outcomes.
- Create and maintain representative datasets from simulations, human annotation, production feedback, and edge cases while protecting sensitive data.
- Evaluate frontier and open-source models and make build, buy, route, prompt, fine-tune, or distill decisions.
- Improve prompting, context construction, memory, tool-use policies, structured outputs, model routing, and fallback behavior.
- Design fine-tuning, preference optimization, distillation, and other post-training work when justified by measurable results.
- Partner with speech and real-time engineers to improve ASR, TTS, turn-taking, interruption handling, pronunciation, multilingual behavior, and end-to-end latency.
- Develop analysis tools for model failures, scenario-based performance slicing, regression detection, and faster iteration.
- Ship model changes with production guardrails, staged rollouts, monitoring, rollback paths, and quality gates.
- Translate research and model releases into product opportunities and communicate tradeoffs across technical and non-technical teams.
Requirements
- At least 3 years of industrial experience in a relevant technical domain.
- Strong machine learning foundations and hands-on experience developing or evaluating production AI systems.
- Strong Python skills and experience with modern ML tooling such as PyTorch, JAX, or Hugging Face.
- Practical experience with LLMs, speech models, multimodal models, or agentic systems.
- Ability to design reliable experiments, define useful metrics, analyze noisy results, and avoid weak proxies.
- Experience building datasets, evaluation harnesses, model services, or training and inference pipelines.
- Strong software engineering judgment, including reproducible, tested, observable, and reusable work.
- Ability to connect modeling decisions to latency, cost, privacy, safety, and user experience constraints.
- Ability to collaborate across research, engineering, product, and customer contexts.
- Curiosity about new model capabilities and limitations.
Benefits
- Competitive salary and stock option plan.
- Medical, dental, vision, retirement, leave, and disability benefits as applicable.
- Family leave.
- Short-term and long-term disability benefits.
- Paid time off and company holidays.
- Learning and development support.
- Full-time, on-site role in Los Altos, United States.