2 days ago
Base Salary
$216k - $281k/yr
Responsibilities
- Own the evaluation roadmap and strategy for agentic AI quality from architecture through production operation.
- Design layered evaluation systems using deterministic checks, model-based evaluation, and calibrated human expert review.
- Build and calibrate multi-model LLM-as-judge panels and validate them against human experts.
- Operate human-in-the-loop evaluation workflows, converting annotations into training signals for automated judges.
- Score live production traffic, detect regressions and drift, and turn production traces into offline test cases.
- Lead simulation, synthetic data generation, adversarial testing, and evaluation of multi-step agent behavior.
- Connect evaluation failures to clustered issues, candidate fixes, and shadow or A/B validation.
- Maintain experiment snapshots, dataset versioning, coverage reporting, and cost tracking for trustworthy results.
- Drive shared standards, data contracts, and judge criteria across partner teams through influence rather than reporting authority.
- Mentor engineers, establish evaluation engineering standards, and represent the domain to senior leadership.
Requirements
- BA/BS in Computer Science, Machine Learning, or a related quantitative field with 8+ years of relevant experience, or an MS with 6+ years, or a PhD with 3+ years.
- Experience taking machine learning or LLM-based systems into production and owning them operationally.
- Experience with Python and modern deep learning frameworks such as PyTorch or TensorFlow.
- Experience designing evaluation, measurement, or quality systems for ML or LLM products, including datasets, metrics, judges, benchmarks, or regression suites.
- Experience setting technical direction across multiple teams through influence rather than reporting lines.
- Preferred experience designing and calibrating LLM-as-judge systems against human experts, including inter-annotator agreement and judge drift.
- Preferred experience evaluating agentic systems, including multi-step reasoning, tool selection, trajectory quality, and root-cause correctness.
- Preferred experience with asynchronous production scoring, drift detection, alerting, simulation, synthetic environments, CI/CD evaluation, cost-aware inference, or networking and observability domains.
- Publications at top ML venues, open-source contributions, or public technical writing in the evaluation space are preferred.
Benefits
- Medical, dental, and vision insurance, a 401(k) with Cisco matching contributions, paid parental leave, disability coverage, and basic life insurance.
- Paid holidays, birthday leave, year-end shutdown, personal wellness days, sick time, vacation or flexible time off depending on employee classification, and optional paid volunteer days.
- Employees may be eligible for Cisco restricted stock units and annual bonuses subject to company policies.
- The posting applies to U.S. and/or Canada locations, with location-specific salary ranges and benefits eligibility governed by Cisco plans and policies.
Categories
About Cisco
Cisco designs and sells networking, security, and collaboration platforms for enterprises, service providers, and governments, spanning routers and switches, Wi‑Fi, firewalls, zero‑trust, observability, and cloud-managed IT (Meraki) plus Webex. Its business model mixes hardware, software subscriptions, and support/consulting services. Founded in 1984 and headquartered in San Jose, California, Cisco is a public company traded on Nasdaq and serves customers across data centers, campuses, and service provider networks.