Senior Machine Learning Engineer, Agent Eval Platform
ServiceNow2 months ago
Responsibilities
- Design shared LLM judges with configurable per-item rubrics for agent evaluations.
- Combine deterministic world-state validators with LLM-based assessment of subjective trajectory qualities.
- Calibrate judge outputs against human-labeled trajectories and maintain confidence-aware routing to human review.
- Fine-tune small judge models and guard against correlated errors between simulators and judges.
- Investigate offline-to-online evaluation divergence and refresh suites with production failures.
- Develop process reward models and use evaluation signals to optimize prompts, tool selection, planner behavior, retrieval, and routing.
- Build versioned scenarios, repeatable simulated environments, and reward infrastructure for future reinforcement learning efforts.
- Ensure step-level scores support diagnosis and training while end-state outcomes remain the definition of agent success.
Requirements
- 5+ years of experience in applied ML, data science, or ML-adjacent engineering with shipped, production-used work.
- Experience converting subjective human judgment into reliable measurements that people or models can act on.
- Strong applied machine learning fundamentals and experience evaluating, prompting, or fine-tuning LLMs rather than pretraining them.
- Strong Python skills and experience shipping production-grade code.
- Experience in at least three relevant areas, including LLM-as-judge evaluation, human annotation programs, search or recommendation evaluation, small-model fine-tuning, reward modeling, RLHF/RLAIF, agent trajectory analysis, or measured prompt engineering.
- Clear communication, strong ownership, comfort with ambiguity, and a bias toward shipping at startup pace.
Benefits
- Regular employee position with a required-in-office work persona in North America and Canada.
- ServiceNow provides flexible work personas based on role and assigned work location, subject to eligibility verification.
- Equal opportunity employer with reasonable accommodation support.
- Employment may require export-control approval for access to controlled technology.
Tech Stack
Categories
About ServiceNow
ServiceNow builds a cloud platform for enterprise digital workflows, covering IT service management, customer service, HR service delivery, security operations, and operations management, plus tools for custom app development. It sells subscription SaaS to large organizations and public-sector agencies to automate processes and connect data across systems. Founded in 2004 and headquartered in Santa Clara, California, ServiceNow is a public company listed on the NYSE under the ticker NOW.