Staff Software Engineer, Agent Eval Platform
ServiceNow2 months ago
Responsibilities
- Build and operate the runtime for end-to-end, multi-turn agent evaluation scenarios, including environments, simulators, traces, validators, scoring, retries, isolation, and teardown.
- Create versioned evaluation specifications, datasets, reports, scheduling workflows, run comparisons, reliability standards, and self-service evaluation capabilities.
- Lead OpenTelemetry-native observability for agent trajectories, including span data models, context propagation, prompt and completion retention, fault attribution, and cross-run comparison.
- Develop stateful enterprise-system simulations with persistent run-time data, hermetic setup and teardown, LLM-driven and scripted simulators, and API contract-testing mocks.
- Lay the foundation for using evaluation signals to optimize and train agents.
Requirements
- 8+ years of experience building production backend or infrastructure systems.
- Strong Python or Go skills, ideally both.
- Experience designing and operating systems handling real traffic at scale.
- Experience in at least three areas including distributed systems, orchestration and workflow runtimes, observability internals, concurrent and asynchronous programming, data-intensive pipelines, and gRPC/protobuf service design.
- Ability to make non-deterministic systems measurable and comfortable working on novel problems with limited established methodology.
Benefits
- Flexible work personas are available in principle, but this position has a Required in Office work persona.
- Regular employee position in the AMS - North America and Canada region.
About ServiceNow
ServiceNow builds a cloud platform for enterprise digital workflows, covering IT service management, customer service, HR service delivery, security operations, and operations management, plus tools for custom app development. It sells subscription SaaS to large organizations and public-sector agencies to automate processes and connect data across systems. Founded in 2004 and headquartered in Santa Clara, California, ServiceNow is a public company listed on the NYSE under the ticker NOW.