14 hours ago
Palo Alto, CA, USAMid Level
Base Salary
$200k - $300k/yr
Responsibilities
- Build autonomous agents that complete delegated work end to end.
- Design evaluation frameworks, success measures, representative datasets, benchmarks, and metrics for real-world agent tasks.
- Develop agent capabilities for tools used by executives and verify readiness before launch.
- Investigate production failures, identify root causes, and deliver preventative fixes.
- Build scalable failure detection using evaluation and monitoring tools.
- Improve agent recovery from tool and website failures.
- Create repeatable workflows to measure whether agent changes improve performance.
Requirements
- At least 3 years of experience in data science or machine learning roles building or delivering evaluation systems, metrics, or product-quality measurement.
- Production-level Python and SQL skills for analysis, large datasets, and automated pipelines.
- Experience designing evaluation frameworks with success criteria, representative datasets, appropriate metrics, and trustworthy benchmarks.
- Experience with statistics, experiment design, and realistic end-to-end simulations for evaluating agent changes.
- Experience building or operating agents for real users and improving their reliability.
- Post-training, fine-tuning, or reinforcement learning experience is valuable.
- Experience owning and shipping systems to users with demonstrated impact.
- Familiarity with Django, React, and TypeScript.
- Demonstrated sustained contribution and growth, such as through role progression, founding a startup, or earning a promotion.
Benefits
- Base salary of $200,000 to $300,000 annually.
- On-site in Palo Alto, California, with four days per week in the office.
- Visa sponsorship is not available.