4 hours ago
Palo Alto, CA, USASenior
Base Salary
$200k - $300k/yr
Responsibilities
- Build agents that complete executive workflows end to end with minimal oversight.
- Define task-success metrics, evaluation frameworks, and representative datasets.
- Develop and validate new agent capabilities before launch.
- Investigate production failures, identify root causes, and deliver preventative fixes.
- Build scalable systems to detect known failure patterns and unexpected issues.
- Improve agent resilience when external tools or websites fail.
- Create repeatable methods for engineers to measure the impact of agent changes.
Requirements
- At least 3 years of experience in data science or machine learning, including evaluation systems, metrics, or product quality measurement.
- Production-quality Python and SQL skills for analysis, large datasets, and automated pipelines.
- Experience designing evaluation frameworks, selecting meaningful metrics, and building representative test datasets.
- Track record delivering production AI, machine learning, agent, or agent-evaluation systems to real users.
- Experience owning systems end to end and applying statistics, experiment design, and realistic simulations.
- Familiarity with post-training, fine-tuning, or reinforcement learning is valuable.
- Experience with Django, React, and TypeScript is preferred; Braintrust or Raindrop exposure is useful.
- Startup experience or sustained growth and promotion at one organization, strong engineering fundamentals, and up to 11 years of experience.
Benefits
- On-site role in Palo Alto, California.
- Visa sponsorship is not available.
