5 days ago
Palo Alto, CA, USAStaff+
Base Salary
$200k - $300k/yr
Responsibilities
- Build autonomous agents that complete delegated executive work end to end.
- Define and measure agent quality using success criteria, datasets, metrics, and benchmarks.
- Develop agent capabilities for tools used by executives and evaluate readiness before launch.
- Investigate production failures, identify root causes, and ship preventative fixes.
- Build detection systems for known failure patterns and unexpected issues at scale.
- Improve agent recovery when tools or websites fail.
- Create repeatable evaluation workflows to measure the impact of engineering changes.
Requirements
- At least 3 years of experience in data science or machine learning roles building or delivering evaluation systems, metrics, or quality measurement.
- Production-quality Python and SQL skills for analysis, large datasets, and automated pipelines.
- Experience designing evaluation frameworks with measurable success criteria, representative datasets, appropriate metrics, and reliable benchmarks.
- A track record of relevant impact, sustained progression in a role, and a degree from a strong engineering program.
- Experience with applied AI or machine learning systems and an ability to collaborate on ambiguous problems.
- Experience with relevant tools and technologies including Django, React, TypeScript, Braintrust, and Raindrop.
Benefits
- Base salary of $200,000 to $300,000 annually.
- On-site in Palo Alto, California, with four days per week in the office.