Software Engineer, Agent Eval Platform
ServiceNow11 hours ago
Base Salary
$114k - $176k/yr
Responsibilities
- Build and operate a runtime for end-to-end, multi-turn agent evaluations, including environment setup, simulation, tracing, validation, scoring, teardown, scheduling, retries, isolation, and run comparison.
- Consolidate one-off evaluations into a reliable, self-service orchestration service with production-scale concurrency, reproducibility, and an SLO for the harness.
- Lead OpenTelemetry-native observability for agent trajectories, including span modeling, context propagation, prompt and completion capture, fault attribution, and cross-run debugging.
- Build stateful simulations and fakes of enterprise systems with per-run data injection, hermetic setup and teardown, deterministic and LLM-driven simulators, and contract-testing mocks.
- Lay the foundation for using evaluation signals to optimize and train the agent.
Requirements
- 0-2 years of experience building production backend or infrastructure systems.
- Strong Python or Go skills, ideally both.
- Experience designing and operating systems that handle real traffic at scale.
- Experience in at least three areas including distributed systems, orchestration and workflow runtimes, observability internals, concurrent and asynchronous programming, data-intensive pipelines, or gRPC/protobuf service design.
- Interest in making non-deterministic agent behavior measurable and comfort working on ambiguous problems without established textbook solutions.
Benefits
- Base pay of $113,500-175,900, plus equity when applicable, variable or incentive compensation, and benefits.
- Health plans including flexible spending accounts, a 401(k) plan with company match, ESPP, matching donations, flexible time away, and family leave programs.
- Regular employee role with a required-in-office work persona in North America and Canada.
Categories
About ServiceNow
ServiceNow builds a cloud platform for enterprise digital workflows, covering IT service management, customer service, HR service delivery, security operations, and operations management, plus tools for custom app development. It sells subscription SaaS to large organizations and public-sector agencies to automate processes and connect data across systems. Founded in 2004 and headquartered in Santa Clara, California, ServiceNow is a public company listed on the NYSE under the ticker NOW.