3 days ago
Base Salary
$200k - $300k/yr
Responsibilities
- Design synthetic company environments with realistic email, records, and files for AI agents.
- Build graders and evaluation systems for advanced AI agents, including cases requiring specialized assessment.
- Develop benchmarks that correlate with real-world production performance.
- Identify and address reward hacking, benchmark gaming, and other evaluation failure modes.
- Simulate inboxes, CRMs, and other software tools with sufficient fidelity for agent interactions at scale.
Requirements
- Hands-on research experience in synthetic data pipelines, benchmarks, evaluations, or model post-training.
- Experience building environments or benchmarks used to train or evaluate agents at a frontier AI lab.
- Experience implementing graders or evaluation systems for AI agents.
- At least one to two years at a fast-growing or AI-focused startup, plus mid-level or senior engineering or research experience.
- Experience simulating software systems for agent tool use and detecting and fixing benchmark flaws and reward hacking.
- A degree from a highly selective university or an equivalent credential, such as founding a company or participating in a recognized founder program.
- High-level competition experience in mathematics, programming, debate, or games is preferred.
Benefits
- Base salary is $200,000 to $300,000 annually, with up to $350,000 for exceptional candidates; additional commission-style bonus and meaningful founding equity are included.
- Visa sponsorship is available.
- On-site five days per week in San Francisco, California.
Categories
AI Research