Senior Software Engineer, Agent Eval Platform
ServiceNow3 months ago
Base Salary
$161k - $274k/yr
Responsibilities
- Build and operate a production-scale orchestration runtime for multi-turn agent scenarios, including environment setup, execution, validation, scoring, retries, scheduling, isolation, and teardown.
- Create versioned evaluation specifications, datasets, reports, run comparisons, and self-service evaluation workflows.
- Lead OpenTelemetry-native observability for agent systems, including span models, context propagation, trace collection, fault attribution, and cross-run debugging.
- Build stateful simulation environments with persistent datastores, per-run data injection, hermetic setup and teardown, user simulators, and API contract-testing mocks.
- Develop the judgement layer for agent evaluation, including rubrics, judges, human-label calibration, and signals for optimizing agent behavior.
- Establish reliability standards and service-level objectives for the evaluation harness.
Requirements
- At least 5 years of experience building production backend or infrastructure systems.
- Experience in at least three areas including distributed systems, workflow orchestration, observability internals, concurrent or asynchronous programming, data-intensive pipelines, and gRPC/protobuf service design.
- Strong Python or Go skills, ideally both.
- Experience designing and operating systems that handle real traffic at scale.
- Ability to make non-deterministic systems measurable and produce signals engineers can use for release gating.
- Comfort working with ambiguity and novel problems without established textbook solutions.
Benefits
- Base pay of $161,300-$274,200, plus equity when applicable, variable or incentive compensation, and benefits.
- Health plans including flexible spending accounts, a 401(k) plan with company match, ESPP, matching donations, flexible time away, and family leave programs.
- Distributed work personas may be flexible, remote, or required in office depending on the role and location; this position is listed as Required in Office.
- Regular employee position in the AMS - North America and Canada region.
Categories
About ServiceNow
ServiceNow builds a cloud platform for enterprise digital workflows, covering IT service management, customer service, HR service delivery, security operations, and operations management, plus tools for custom app development. It sells subscription SaaS to large organizations and public-sector agencies to automate processes and connect data across systems. Founded in 2004 and headquartered in Santa Clara, California, ServiceNow is a public company listed on the NYSE under the ticker NOW.