2 days ago
Remote, Canada or Vancouver, CanadaStaff+
Responsibilities
- Set the technical direction for the Copilot offline evaluation platform, including scenarios, metrics, and scorecards.
- Architect AI-first and agentic-first services, pipelines, tooling, observability, and guardrails that autonomous agents can operate and extend.
- Design for dependable results across complex service dependencies under heavy and unusual load using graceful degradation, intelligent retries, and dependency-aware gating.
- Define evaluation methods for agentic experiences, including multi-turn trajectories, tool selection, end-to-end task outcomes, UX, and voice modalities.
- Build simulation, scraping, and scoring capabilities to make agent behavior measurable.
- Modernize and scale the evaluation runtime and platform while improving cost efficiency and latency.
- Mentor engineers, lead by example, and share emerging practices in agentic systems, AI evaluation, and autonomous operations.
Requirements
- Bachelor’s degree in Computer Science or a related technical field and 6+ years of technical engineering experience with coding in C, C++, C#, Java, JavaScript, or Python, or equivalent experience.
- Ability to meet Microsoft, customer, and/or government security screening requirements, including the Microsoft Cloud Background Check.
- Preferred: Master’s degree and 8+ years of technical engineering experience, or bachelor’s degree and 12+ years of experience, or equivalent experience.
- Deep experience with large-scale distributed systems and reliability engineering across interacting service dependencies and heavy or unusual load.
- Experience building AI-first or agentic systems and designing APIs, tooling, observability, guardrails, and safety mechanisms for autonomous-agent operation.
- Experience with AI/ML evaluation, measurement, or benchmarking systems, including LLM-based grading, metric design, or large-scale data and quality pipelines.
Benefits
- Typical base pay range is CAD $142,400–$257,500 per year across Canada.
- The position is open for at least five days, with applications accepted continuously until filled.
- Employment is subject to required Microsoft Cloud background screening upon hire or transfer and every two years thereafter.
Tech Stack
Categories
About Microsoft
Microsoft develops operating systems, productivity software, cloud services, developer tools, and consumer devices for individuals, enterprises, and governments. Its main products include Windows, Microsoft 365, Azure, Visual Studio/GitHub, Xbox, and LinkedIn; revenue comes from software subscriptions and licenses, cloud consumption, hardware sales, and advertising. Founded in 1975 and headquartered in Redmond, Washington, Microsoft is a public company traded on Nasdaq.