Staff Machine Learning Engineer, Agent Eval Platform
ServiceNow7 days ago
Responsibilities
- Design shared LLM judges with per-item rubrics configured alongside evaluation datasets.
- Separate deterministic world-state validators from LLM-based evaluation of subjective agent behavior.
- Build confidence-aware scoring and calibrate judges against human-labeled trajectories in partnership with annotation teams.
- Fine-tune small judge models when off-the-shelf models are insufficient.
- Identify correlated blind spots between user simulators and judges and investigate offline-versus-online evaluation divergence.
- Maintain suites reseeded from production failures to avoid overfitting.
- Develop process reward models and use evaluation signals to optimize prompts, tool selection, planner behavior, retrieval, and routing.
- Build versioned scenarios, repeatable simulated worlds, and reward infrastructure for future reinforcement-learning efforts.
- Ensure step-level scores support diagnosis and training while end-state outcomes remain the measure of agent success.
Requirements
- Have 8+ years of experience in applied ML, data science, or ML-adjacent engineering with shipped and adopted work.
- Have experience converting subjective human judgment into reliable measurements that people or models can act on.
- Demonstrate strong applied ML fundamentals and experience evaluating, prompting, or fine-tuning LLMs rather than pretraining them.
- Write strong Python and production-grade code beyond notebook-based work.
- Have experience in at least three areas including LLM-as-judge evaluation, human annotation programs, search or recommendation evaluation, small-model fine-tuning, reward modeling, RLHF/RLAIF, agent trajectory analysis, or measured prompt engineering.
- Communicate clearly about complex measurement problems and exercise ownership, judgment, and comfort with ambiguity.
Benefits
- Regular employee position with a required-in-office work persona.
- Work-persona framework supports flexible, remote, or required-in-office arrangements depending on role and location, with this position designated required in office.
- Role is part of a distributed work environment backed by ServiceNow's global scale.
Tech Stack
Categories
About ServiceNow
ServiceNow (NYSE: NOW) makes the world work better for everyone. Our cloud-based platform and solutions help digitize and unify organizations so that they can find smarter, faster, better ways to make work flow. So employees and customers can be more connected, more innovative, and more agile. And we can all create the future we imagine. The world works with ServiceNow. For more information, visit www.servicenow.com.