4 hours ago
Base Salary
$150k - $300k/yr
Responsibilities
- Design synthetic company environments with realistic email, records, and files for AI agents.
- Build graders and evaluation systems for assessing agent performance, including performance beyond human reviewers' expertise.
- Develop benchmarks reflecting real-world production performance.
- Identify and fix evaluation failures, reward hacking, and benchmark gaming.
- Simulate software tools such as inboxes and CRMs to support realistic agent interactions at scale.
Requirements
- Experience building environments or benchmarks used by frontier AI labs to train or evaluate agents.
- Hands-on experience designing and implementing AI agent graders or evaluation systems.
- Experience simulating software systems that withstand agent tool use and building synthetic environments for production-like evaluation.
- Track record of identifying flaws in widely used benchmarks and detecting and fixing reward hacking in graders.
- Startup or early-stage company experience and strong competitive problem-solving experience in areas such as math, programming, debate, or games.
- Collaborative approach to ambiguous, fast-paced work and strong conviction in AI agents' potential to perform computer-based work.
Benefits
- Meaningful founding equity.
- On-site five days a week in San Francisco, California.
- Visa sponsorship availability requires confirmation because the supplied details differ.