3 days ago
Vancouver, CanadaSenior
Responsibilities
- Build, evaluate, and improve AI/ML experiences involving LLMs, RAG, agentic workflows, and multimodal systems.
- Design evaluation frameworks for quality, grounding, reliability, safety, and user experience.
- Build offline experimentation pipelines for models, prompts, retrieval, and system designs.
- Partner with product, engineering, and science teams to define measurable objectives and success metrics.
- Analyze failures and drive product and technical improvements.
- Build scalable, reliable, observable, maintainable production ML systems.
- Contribute to MLOps through data pipelines, telemetry, regression testing, and monitoring.
- Communicate risks, tradeoffs, and recommendations while mentoring peers.
Requirements
- Bachelor's degree in a relevant field and 4+ years of related experience, or a master's degree and 3+ years, or a doctorate and 1+ year, or equivalent experience.
- Ability to meet Microsoft, customer, and government security screening requirements, including the Microsoft Cloud Background Check.
- Preferred qualifications include a master's degree with 6+ years or a doctorate with 3+ years of related experience, or equivalent experience.
- At least 3 years of experience creating publications, patents, or libraries; conducting research; or equivalent recognized technical contributions, as applicable.
- Experience presenting research or technical work at conferences or industry events.
- Experience developing and deploying live production systems and products across multiple stages of the product lifecycle.
- Experience building, evaluating, and shipping production AI/ML systems, including LLM-based applications.
- Hands-on experience with prompting, evaluation, RAG, agents and tool use, failure analysis, experimentation, error analysis, and data-driven improvement.
- Understanding of evaluation methods, dataset and annotation strategy, benchmarks, regression testing, and production monitoring.
- Strong software engineering, product judgment, communication, and cross-functional collaboration skills.
Categories
About Microsoft
Microsoft develops operating systems, productivity software, cloud services, developer tools, and consumer devices for individuals, enterprises, and governments. Its main products include Windows, Microsoft 365, Azure, Visual Studio/GitHub, Xbox, and LinkedIn; revenue comes from software subscriptions and licenses, cloud consumption, hardware sales, and advertising. Founded in 1975 and headquartered in Redmond, Washington, Microsoft is a public company traded on Nasdaq.