3 days ago
Base Salary
$200k - $300k/yr
Responsibilities
- Design synthetic company environments with realistic email, records, and files for extended AI-agent use.
- Build graders and evaluation systems capable of assessing AI agents beyond human reviewer expertise.
- Develop benchmarks that correlate with real-world production performance.
- Identify and fix evaluation failures, including reward hacking and benchmark gaming.
- Simulate software tools such as inboxes and CRMs with sufficient fidelity for agent interactions at scale.
Requirements
- Hands-on experience with synthetic data pipelines, benchmarks, evaluations, or post-training in an AI research or engineering context.
- Experience building environments or benchmarks used to train or evaluate AI agents and designing agent graders or evaluation systems.
- Experience simulating complex software systems for agent tool use and identifying benchmark flaws or fixing reward hacking.
- At least 1 to 2 years at a fast-growing or AI-focused startup with relevant AI research or engineering experience.
- Strong startup experience, such as founding a company or participating in an accelerator.
- High-level competition experience in areas such as mathematics, programming, debate, or games.
- Conviction that AI agents can outperform humans at many computer-based tasks.
Benefits
- $200,000 to $300,000 annual base salary, with up to $350,000 for exceptional candidates.
- Commission-style bonus of up to $200,000 or more tied to the value of delivered environments.
- Meaningful founding equity is available.
- Visa sponsorship is available.
- On-site five days a week in San Francisco, California.