Principal AI Architect - M365 IC3 Team (Intelligent Conversation and Communications Cloud)
Microsoft13 hours ago
Base Salary
$143k - $304k/yr
Responsibilities
- Define the technical vision and architecture for AI, LLM, RAG, and agent evaluation systems across the product lifecycle.
- Design evaluation frameworks for development, launch, and post-release quality measurement.
- Analyze nondeterministic agent behavior, failure modes, model limitations, retrieval issues, orchestration defects, and product-quality risks.
- Determine whether issues should be addressed in models, prompts, tools, retrieval, ranking, orchestration, UX, policy, telemetry, or product code.
- Integrate evaluations into build pipelines, release gates, experimentation systems, dashboards, and engineering workflows.
- Connect telemetry, offline evaluation, human judgment, automated evaluations, experimentation, RAG quality, agent behavior, and customer-quality signals.
- Evaluate enterprise search, grounding, indexing, ranking, permissions, freshness, and relevance systems for products such as Copilot.
- Drive architecture decisions across services, data flows, model interfaces, search systems, retrieval layers, evaluation harnesses, dashboards, and reporting systems.
- Translate ambiguous product goals into measurable evaluation strategies, success criteria, timelines, and technical plans.
- Partner with product, engineering, applied science, and data science leaders to prioritize evaluation investments and release decisions.
- Mentor senior engineers and applied scientists on reliable, scalable, reusable evaluation infrastructure.
- Stay current with LLM evaluation, agentic systems, RAG evaluation, benchmark design, experimentation, and responsible AI practices.
Requirements
- Bachelor's degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or a related field and 6+ years of related experience; alternatively, a master's degree and 4+ years, a doctorate and 3+ years, or equivalent experience.
- Preferred qualifications include a master's degree and 9+ years or a doctorate and 6+ years of related experience, or equivalent experience.
- 5+ years of experience creating publications such as patents, libraries, or peer-reviewed academic papers.
- 2+ years of experience presenting at conferences or external research or industry events as an invited speaker.
- 5+ years of experience conducting research in academic or industry settings.
- 3+ years of experience developing and deploying live production systems as part of a product team.
- 3+ years of experience developing and deploying products or systems across multiple points in the product lifecycle.
- Demonstrated experience evaluating LLMs through evaluation datasets, quality metrics, model-behavior analysis, failure-mode identification, and product improvement.
- Experience with modern AI systems, agentic systems, RAG, ML pipelines, evaluation methodology, experimentation, and product telemetry.
- Experience architecting complex systems involving multiple components, services, models, data flows, tools, retrieval systems, and product surfaces.
- Experience bringing ML, data science, LLM, or applied science concepts into production systems and product codebases.
- Experience with offline evaluations, online experimentation, human evaluation, red teaming, synthetic data, model monitoring, RAG evaluation, and agent behavior analysis.
- Experience with evaluation dashboards, scorecards, quality reporting, product-health monitoring, or data visualization systems.
- Familiarity with responsible AI, safety, reliability, privacy, security, permissions, compliance, and enterprise-readiness considerations.
- Experience building evaluation platforms, experimentation systems, model observability, agent evaluation infrastructure, or product-quality infrastructure.
- Experience operating at a principal, architect, or senior technical leadership level.
- Strong coding, technical design, communication, cross-functional influence, and ambiguous-initiative leadership skills.
Benefits
- The typical U.S. base pay range is USD $142,800-$274,800 per year, with a separate San Francisco Bay Area and New York City metropolitan area range of USD $188,000-$304,200 per year.
- Certain roles may be eligible for benefits and other compensation.
- The position will remain open for a minimum of 5 days, with applications accepted on an ongoing basis until filled.
Categories
About Microsoft
Microsoft develops operating systems, productivity software, cloud services, developer tools, and consumer devices for individuals, enterprises, and governments. Its main products include Windows, Microsoft 365, Azure, Visual Studio/GitHub, Xbox, and LinkedIn; revenue comes from software subscriptions and licenses, cloud consumption, hardware sales, and advertising. Founded in 1975 and headquartered in Redmond, Washington, Microsoft is a public company traded on Nasdaq.