Senior Machine Learning Engineer, Agent Eval Platform
ServiceNow7 days ago
Responsibilities
- Design shared LLM judges with configurable per-item rubrics for agent evaluations.
- Combine deterministic world-state validators with LLM-based assessment of subjective trajectory qualities.
- Calibrate judge outputs against human-labeled trajectories and maintain confidence-aware routing to human review.
- Fine-tune small judge models and guard against correlated errors between simulators and judges.
- Investigate offline-to-online evaluation divergence and refresh suites with production failures.
- Develop process reward models and use evaluation signals to optimize prompts, tool selection, planner behavior, retrieval, and routing.
- Build versioned scenarios, repeatable simulated environments, and reward infrastructure for future reinforcement learning efforts.
- Ensure step-level scores support diagnosis and training while end-state outcomes remain the definition of agent success.
Requirements
- 5+ years of experience in applied ML, data science, or ML-adjacent engineering with shipped, production-used work.
- Experience converting subjective human judgment into reliable measurements that people or models can act on.
- Strong applied machine learning fundamentals and experience evaluating, prompting, or fine-tuning LLMs rather than pretraining them.
- Strong Python skills and experience shipping production-grade code.
- Experience in at least three relevant areas, including LLM-as-judge evaluation, human annotation programs, search or recommendation evaluation, small-model fine-tuning, reward modeling, RLHF/RLAIF, agent trajectory analysis, or measured prompt engineering.
- Clear communication, strong ownership, comfort with ambiguity, and a bias toward shipping at startup pace.
Benefits
- Regular employee position with a required-in-office work persona in North America and Canada.
- ServiceNow provides flexible work personas based on role and assigned work location, subject to eligibility verification.
- Equal opportunity employer with reasonable accommodation support.
- Employment may require export-control approval for access to controlled technology.
Tech Stack
Categories
About ServiceNow
ServiceNow (NYSE: NOW) makes the world work better for everyone. Our cloud-based platform and solutions help digitize and unify organizations so that they can find smarter, faster, better ways to make work flow. So employees and customers can be more connected, more innovative, and more agile. And we can all create the future we imagine. The world works with ServiceNow. For more information, visit www.servicenow.com.