Location
Redmond, WA, United States
Posted
July 27, 2026
Job Description
**Overview**
How do you know an AI product is actually getting better, and how do you prove it, at scale, before millions of people feel the difference? We build and operate the offline evaluation platform that gates Microsoft Copilot’s quality: teams across Copilot depend on us to run their scenarios against the product, score the responses, and produce the scorecards that decide what ships. We do this at the scale of one of the world’s largest AI products, inside a strict enterprise compliance boundary. As Copilot becomes a fleet of autonomous agents, the way we measure quality has to be reinvented, and you will lead that reinvention.
As a Principal Software Engineer on the Copilot evaluation team, you will own the technical vision for a platform built AI-first and agentic-first: one where autonomous agents are not just measured, but help operate, extend, and heal the system itself. You will take on our hardest problem, delivering trustworthy results across a deep c...
How do you know an AI product is actually getting better, and how do you prove it, at scale, before millions of people feel the difference? We build and operate the offline evaluation platform that gates Microsoft Copilot’s quality: teams across Copilot depend on us to run their scenarios against the product, score the responses, and produce the scorecards that decide what ships. We do this at the scale of one of the world’s largest AI products, inside a strict enterprise compliance boundary. As Copilot becomes a fleet of autonomous agents, the way we measure quality has to be reinvented, and you will lead that reinvention.
As a Principal Software Engineer on the Copilot evaluation team, you will own the technical vision for a platform built AI-first and agentic-first: one where autonomous agents are not just measured, but help operate, extend, and heal the system itself. You will take on our hardest problem, delivering trustworthy results across a deep c...