3 days ago
Palo Alto, CA, USAMid Level
Base Salary
$200k - $300k/yr
Responsibilities
- Build autonomous agents that complete delegated work end to end for real users.
- Define success measures, evaluation frameworks, representative datasets, and metrics for agent tasks.
- Develop and validate agent capabilities before release.
- Investigate production failures and implement fixes addressing underlying causes.
- Build systems to detect known failure patterns and emerging issues at scale.
- Improve agent recovery when tools or websites fail.
- Create repeatable methods for measuring the production impact of agent changes.
Requirements
- At least 3 years of data science or machine learning experience building evaluation systems, metrics, or quality measurement for products.
- Production-quality Python and SQL skills, including analysis of large datasets and automated pipeline development.
- Experience designing evaluation frameworks, datasets, measurable criteria, and appropriate metrics.
- Track record delivering production AI, machine learning, agent, or agent-evaluation systems for real users.
- Experience improving agent reliability, owning systems end to end, and measuring results in production.
- Familiarity with statistics, experiment design, and realistic end-to-end simulations.
- Experience with Python, Django, React, and TypeScript; familiarity with Braintrust or Raindrop is useful.
- Fine-tuning, post-training, or reinforcement learning experience is valuable.
- Demonstrated progression and sustained experience in a role, with the ability to work through ambiguity and ship quickly.
Benefits
- Base salary of $200,000 to $300,000 USD annually.
- On-site role in Palo Alto, California, with four days per week in the office.
- Visa sponsorship is not available.