Senior Software Engineer, Agent Eval Platform
ServiceNow21 days ago
Base Salary
$161k - $274k/yr
Responsibilities
- Build the judgement layer for agent evaluation, including rubrics, LLM judges, calibration against human labels, and meaningful scoring methodology.
- Develop and operate orchestration services for multi-turn agent scenarios, including scheduling, retries, isolation, versioning, comparison, and self-service execution.
- Lead OpenTelemetry-native observability for agent trajectories, including span models, context propagation, prompt and completion capture, fault attribution, and debugging surfaces.
- Build stateful, hermetic simulations of enterprise systems with data injection, setup and teardown, user simulators, and contract-testing mocks.
- Establish reliability standards and an SLO for the evaluation harness and lay the foundation for using evaluation signals to optimize agents.
Requirements
- 5+ years building production backend or infrastructure systems.
- Strong proficiency in Python or Go, ideally both.
- Experience designing and operating systems that handle real traffic at scale.
- Experience in at least three areas: distributed systems; orchestration and workflow runtimes; observability internals; concurrent and asynchronous programming; data-intensive pipelines; or gRPC/protobuf service and interface design.
- Ability to make non-deterministic systems measurable and turn agent behavior into signals suitable for release gating.
- Comfort working through ambiguity on novel problems without established textbook solutions.
Benefits
- Base pay of $161,300-$274,200, plus equity when applicable and variable or incentive compensation.
- Health plans including flexible spending accounts, a 401(k) plan with company match, ESPP, matching donations, flexible time away, and family leave programs.
- Regular employee position with a required-in-office work persona in the North America and Canada region.
Categories
About ServiceNow
ServiceNow (NYSE: NOW) makes the world work better for everyone. Our cloud-based platform and solutions help digitize and unify organizations so that they can find smarter, faster, better ways to make work flow. So employees and customers can be more connected, more innovative, and more agile. And we can all create the future we imagine. The world works with ServiceNow. For more information, visit www.servicenow.com.