18 hours ago
San Francisco, CA, USAEntry Level / Mid Level
Base Salary
$200k - $350k/yr
Responsibilities
- Design synthetic company environments with realistic emails, records, and files.
- Build graders and evaluation systems for assessing AI agent performance.
- Develop benchmarks that correlate with real-world production performance.
- Identify and address reward hacking, benchmark gaming, and other evaluation failure modes.
- Simulate inboxes, CRMs, and other tools with sufficient fidelity for agent interactions at scale.
Requirements
- Hands-on research experience with synthetic data pipelines, benchmarks, evaluations, or post-training.
- At least 1 to 2 years of experience at a fast-growing or AI-focused startup.
- Ability to simulate complex software systems that can withstand agent tool use.
- Experience building environments or benchmarks for frontier AI training or evaluation is valuable.
- A track record of finding flaws in established benchmarks and improving evaluation systems is valued.
- Startup experience, strong performance in competitive mathematics, programming, debate, or games, and collaborative performance under pressure are valued.
Benefits
- Meaningful founding equity is available.
- This is an on-site role five days per week in San Francisco, California.