2 days ago
Base Salary
$120k - $261k/yr
Responsibilities
- Lead applied science strategy for AI-powered product areas from opportunity identification through experimentation, launch, and continuous improvement.
- Define customer, quality, reliability, safety, engagement, and business-impact measures for AI product experiences.
- Design and operationalize LLM evaluation frameworks across prototyping, development, launch validation, and production monitoring.
- Build evaluation methodologies and offline experimentation pipelines for LLM-based, agentic, retrieval-augmented, and multimodal AI systems.
- Partner with engineering and product leadership on strategy, roadmaps, customer needs, reliability, scalability, latency, observability, maintainability, and MLOps.
- Drive hypothesis formation, metric definition, experimentation, result interpretation, and product or technical recommendations.
- Establish evaluation scorecards, regression testing, guardrail metrics, launch criteria, and responsible AI decision frameworks.
- Mentor scientists and engineers, influence cross-team practices, and communicate findings, risks, tradeoffs, and recommendations.
Requirements
- Bachelor's degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or a related field and 6+ years of related experience; or a master's degree and 4+ years; or a doctorate and 3+ years; or equivalent experience.
- Significant experience building, evaluating, and shipping AI/ML-powered products or services.
- Hands-on experience with large language models, prompt engineering, RAG, agents, tool use, model comparison, failure analysis, and evaluation-driven iteration.
- Deep understanding of LLM evaluation, including dataset creation, annotation, automated and human evaluation, benchmark design, regression testing, launch criteria, and production monitoring.
- Experience designing offline experiments and interpreting results for model, system, and product decisions.
- Experience evaluating multimodal AI systems or experiences with multiple input/output modalities.
- Understanding of product success metrics and ability to translate customer and business goals into measurable scientific and technical objectives.
- Solid software engineering skills and experience collaborating on production-quality AI systems.
- Ability to meet Microsoft, customer, and/or government security screening requirements, including the Microsoft Cloud background check.
- Preferred qualifications include a PhD or doctorate in a relevant field with the stated related experience.
Benefits
- Typical U.S. base pay range is USD $119,800-$234,700 per year, with a separate USD $160,200-$261,000 annual range for specified San Francisco Bay Area and New York City metropolitan area locations.
- Certain roles may be eligible for benefits and other compensation.
- The position will remain open for at least 5 days and accepts applications on an ongoing basis until filled.
Categories
About Microsoft
Microsoft develops operating systems, productivity software, cloud services, developer tools, and consumer devices for individuals, enterprises, and governments. Its main products include Windows, Microsoft 365, Azure, Visual Studio/GitHub, Xbox, and LinkedIn; revenue comes from software subscriptions and licenses, cloud consumption, hardware sales, and advertising. Founded in 1975 and headquartered in Redmond, Washington, Microsoft is a public company traded on Nasdaq.