2 hours ago
Palo Alto, CA, USAMid Level
Base Salary
$200k - $300k/yr
Responsibilities
- Build autonomous agents that complete delegated work end to end without requiring users to check each step.
- Define measurable success criteria and evaluation methods for agent performance across tasks.
- Develop and evaluate agent capabilities before releasing them to users.
- Investigate production failures, identify underlying causes, and deliver durable fixes.
- Build systems using Braintrust and Raindrop to detect known and unpredictable failure patterns at scale.
- Improve agent recovery when tools or websites fail.
- Create repeatable evaluation workflows for measuring the impact of engineering changes.
Requirements
- At least 3 years of data science or machine learning experience building evaluation systems, metrics, or product-quality measurement.
- Production-quality Python and SQL skills for analysis, large datasets, and automated pipelines.
- Experience designing evaluation frameworks, representative datasets, meaningful metrics, and realistic end-to-end simulations.
- Track record delivering production AI, ML, agent, or agent-evaluation systems for real users and measuring their impact.
- Experience improving agent reliability, owning systems end to end, and using statistics and experiment design to evaluate changes.
- Experience with Django, React, and TypeScript.
- Familiarity with post-training, fine-tuning, or reinforcement learning is useful.
- Demonstrated progression and sustained experience in a role, with comfort working in a fast-moving, ambiguous environment.
Benefits
- On-site work in Palo Alto, California.
- Visa sponsorship is not available.