Tech Lead, Agent Eval Platform
ServiceNow2 months ago
Responsibilities
- Lead engineers and own end-to-end delivery of agent evaluation platform projects.
- Build orchestration services for multi-turn agent scenarios, including environment setup, simulation, validation, scoring, retries, isolation, scheduling, and teardown.
- Develop versioned datasets, specifications, reports, run comparisons, and self-service evaluation workflows.
- Lead the transition to OpenTelemetry-native observability with queryable trajectory spans, context propagation, prompt and completion capture, fault attribution, and cross-run diffing.
- Build stateful simulation environments with persistent datastores, hermetic setup and teardown, LLM-driven user simulators, deterministic state-machine simulators, and API contract-testing mocks.
- Establish reliability floors and SLOs for the evaluation harness.
- Lay the foundation for using evaluation signals to optimize and train agent behavior.
Requirements
- 10+ years building production backend or infrastructure systems.
- Strong proficiency in Python or Go, ideally both.
- Experience designing and operating systems that handle real traffic at scale.
- Experience in at least three areas including distributed systems, orchestration and workflow runtimes, observability internals, concurrent and asynchronous programming, data-intensive pipelines, and gRPC/protobuf service design.
- Ability to technical-lead other engineers and own end-to-end project delivery.
- Comfort making nondeterministic systems measurable and working through ambiguous problems without established textbook solutions.
- Experience with scheduling, retries, high-concurrency systems, trace data, data pipelines, or service and interface design is relevant to the role.
Benefits
- Regular employee position.
- Required in-office work persona in the AMS - North America and Canada region.
- ServiceNow/Moveworks distributed-work approach includes flexible, remote, or required-in-office personas depending on role and location.
- Equal opportunity employer with reasonable accommodation support.
Categories
About ServiceNow
ServiceNow builds a cloud platform for enterprise digital workflows, covering IT service management, customer service, HR service delivery, security operations, and operations management, plus tools for custom app development. It sells subscription SaaS to large organizations and public-sector agencies to automate processes and connect data across systems. Founded in 2004 and headquartered in Santa Clara, California, ServiceNow is a public company listed on the NYSE under the ticker NOW.