10 hours ago
Base Salary
$120k - $261k/yr
Responsibilities
- Lead applied science strategy for AI-powered product areas from opportunity identification through experimentation, launch, and continuous improvement.
- Define customer, quality, reliability, safety, engagement, and business success metrics for AI product experiences.
- Design and operationalize LLM evaluation frameworks across prototyping, development, launch validation, and post-launch monitoring.
- Build evaluation methodologies and offline experimentation pipelines for LLM, agentic, retrieval-augmented, and multimodal AI systems.
- Evaluate models, prompts, retrieval strategies, grounding quality, tool use, and end-to-end product behavior.
- Partner with engineering and product leadership to align science priorities with product strategy and roadmap decisions.
- Collaborate with software engineers to build reliable, scalable, observable, maintainable, production-quality AI systems using MLOps practices.
- Establish evaluation scorecards, regression testing, guardrail metrics, launch criteria, and responsible AI decision frameworks.
- Mentor scientists and engineers and communicate findings, tradeoffs, risks, and recommendations to technical and non-technical stakeholders.
Requirements
- Bachelor's degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or a related field plus 4+ years of related experience; alternatively, a master's degree plus 3+ years, a doctorate plus 1+ year, or equivalent experience.
- Ability to meet Microsoft, customer, and government security screening requirements, including the Microsoft Cloud Background Check.
- Significant experience building, evaluating, and shipping AI/ML-powered products or services.
- Hands-on experience with large language models, prompt engineering, retrieval-augmented generation, agents, tool use, model comparison, failure analysis, and evaluation-driven iteration.
- Deep understanding of LLM evaluation across the development lifecycle, including datasets, annotation, automated and human evaluation, benchmarks, regression testing, launch criteria, and production monitoring.
- Experience designing offline experiments and interpreting results for model, system, and product decisions.
- Experience evaluating multimodal AI systems or experiences with multiple input and output modalities.
- Understanding of product success metrics and the ability to translate customer and business goals into measurable scientific and technical objectives.
- Solid software engineering skills and experience collaborating with engineers on production-quality AI systems.
- Preferred qualifications include a PhD or doctorate with 3-6+ years of related experience or equivalent experience.
Categories
About Microsoft
Microsoft develops operating systems, productivity software, cloud services, developer tools, and consumer devices for individuals, enterprises, and governments. Its main products include Windows, Microsoft 365, Azure, Visual Studio/GitHub, Xbox, and LinkedIn; revenue comes from software subscriptions and licenses, cloud consumption, hardware sales, and advertising. Founded in 1975 and headquartered in Redmond, Washington, Microsoft is a public company traded on Nasdaq.