Staff Software Engineer, Agent Eval Platform
ServiceNow7 days ago
Responsibilities
- Build and operate the runtime for end-to-end, multi-turn agent evaluation scenarios, including environments, simulators, traces, validators, scoring, retries, isolation, and teardown.
- Create versioned evaluation specifications, datasets, reports, scheduling workflows, run comparisons, reliability standards, and self-service evaluation capabilities.
- Lead OpenTelemetry-native observability for agent trajectories, including span data models, context propagation, prompt and completion retention, fault attribution, and cross-run comparison.
- Develop stateful enterprise-system simulations with persistent run-time data, hermetic setup and teardown, LLM-driven and scripted simulators, and API contract-testing mocks.
- Lay the foundation for using evaluation signals to optimize and train agents.
Requirements
- 8+ years of experience building production backend or infrastructure systems.
- Strong Python or Go skills, ideally both.
- Experience designing and operating systems handling real traffic at scale.
- Experience in at least three areas including distributed systems, orchestration and workflow runtimes, observability internals, concurrent and asynchronous programming, data-intensive pipelines, and gRPC/protobuf service design.
- Ability to make non-deterministic systems measurable and comfortable working on novel problems with limited established methodology.
Benefits
- Flexible work personas are available in principle, but this position has a Required in Office work persona.
- Regular employee position in the AMS - North America and Canada region.
About ServiceNow
ServiceNow (NYSE: NOW) makes the world work better for everyone. Our cloud-based platform and solutions help digitize and unify organizations so that they can find smarter, faster, better ways to make work flow. So employees and customers can be more connected, more innovative, and more agile. And we can all create the future we imagine. The world works with ServiceNow. For more information, visit www.servicenow.com.