1 day ago
Base Salary
$120k - $261k/yr
Responsibilities
- Lead applied science strategy for AI-powered product areas from opportunity identification through experimentation, launch, and continuous improvement.
- Define customer value, quality, reliability, safety, engagement, and business-impact success metrics for AI product experiences.
- Design and operationalize LLM evaluation frameworks across prototyping, model and prompt iteration, feature development, launch validation, and post-launch monitoring.
- Build evaluation methodologies for LLM-based, agentic, retrieval-augmented, and multimodal systems involving text, image, document, audio, video, and interaction-based experiences.
- Develop offline experimentation and evaluation pipelines for models, prompts, system designs, retrieval strategies, grounding, tool use, and end-to-end product behavior.
- Partner with engineering and product leadership on science priorities, product strategy, roadmap decisions, customer needs, and execution plans.
- Collaborate with software engineers to build reliable, scalable, observable, maintainable, and production-quality AI systems using MLOps practices.
- Run hypothesis-driven experiments, interpret results, and recommend product or technical actions.
- Establish evaluation scorecards, regression testing, guardrail metrics, launch criteria, and responsible AI decision frameworks.
- Mentor scientists and engineers, influence cross-team AI practices, and communicate findings, risks, tradeoffs, and recommendations to technical and non-technical stakeholders.
Requirements
- Bachelor's degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or a related field plus 6+ years of related experience; or a master's degree plus 4+ years; or a doctorate plus 3+ years; or equivalent experience.
- Significant experience building, evaluating, and shipping AI/ML-powered products or services.
- Hands-on experience with large language models, prompt engineering, retrieval-augmented generation, agents, tool use, model comparison, failure analysis, and evaluation-driven iteration.
- Deep experience with LLM evaluation, including dataset creation, annotation strategies, automated and human evaluation, benchmark design, regression testing, launch criteria, and production monitoring.
- Experience designing offline experiments and using results to guide model, system, and product decisions.
- Experience evaluating multimodal AI systems or experiences involving multiple input and output modalities.
- Understanding of product success metrics and ability to translate customer and business goals into measurable scientific and technical objectives.
- Solid software engineering skills and experience collaborating with engineers on production-quality AI systems.
- Preferred qualifications include a PhD or doctorate in a relevant field with 3-6+ years of related experience or equivalent experience.
Categories
About Microsoft
Microsoft develops operating systems, productivity software, cloud services, developer tools, and consumer devices for individuals, enterprises, and governments. Its main products include Windows, Microsoft 365, Azure, Visual Studio/GitHub, Xbox, and LinkedIn; revenue comes from software subscriptions and licenses, cloud consumption, hardware sales, and advertising. Founded in 1975 and headquartered in Redmond, Washington, Microsoft is a public company traded on Nasdaq.