Base Salary
$143k - $275k/yr
Responsibilities
- Set the technical direction for the Copilot offline evaluation platform and convert ambiguous quality questions into rigorous, reproducible evaluations.
- Lead architecture for an AI-first, agentic-first platform with services, pipelines, tooling, observability, and guardrails that autonomous agents can operate and extend.
- Design for trustworthy results across many dependent services under sustained and unusual load through graceful degradation, intelligent retries, dependency-aware gating, and reliability mechanisms.
- Define end-to-end evaluation methods for agentic Copilot experiences, including multi-turn trajectories, tool selection, task outcomes, UX, and voice.
- Mentor engineers, drive modernization of the evaluation runtime, and improve platform scalability, cost efficiency, and latency.
- Track emerging practices in agentic systems, large-scale AI evaluation, and autonomous operations and share knowledge across the organization.
Requirements
- Bachelor’s degree in Computer Science or a related technical field and 6+ years of technical engineering experience coding in languages such as C, C++, C#, Java, JavaScript, or Python, or equivalent experience.
- Ability to meet Microsoft, customer, and/or government security screening requirements, including the Microsoft Cloud Background Check.
- Preferred: master’s degree and 8+ years of technical engineering experience, or bachelor’s degree and 12+ years of experience, or equivalent experience.
- Deep experience with large-scale distributed systems and reliability engineering across interacting service dependencies under heavy or unusual load.
- Experience building AI-first or agentic systems and designing APIs, tooling, observability, guardrails, and safety mechanisms for autonomous-agent operation.
- Experience with AI/ML evaluation, measurement, or benchmarking systems, including LLM-based grading, metric design, or large-scale data and quality pipelines.
Tech Stack
Categories
About Microsoft
Every company has a mission. What's ours? To empower every person and every organization to achieve more. We believe technology can and should be a force for good and that meaningful innovation contributes to a brighter world in the future and today. Our culture doesn’t just encourage curiosity; it embraces it. Each day we make progress together by showing up as our authentic selves. We show up with a learn-it-all mentality. We show up cheering on others, knowing their success doesn't diminish our own. We show up every day open to learning our own biases, changing our behavior, and inviting in differences. Because impact matters. Microsoft operates in 190 countries and is made up of approximately 228,000 passionate employees worldwide.