6 hours ago
Base Salary
$200k - $300k/yr
Responsibilities
- Design synthetic company environments with realistic email, records, and files for agents to use over extended tasks.
- Build graders and evaluation systems for advanced AI agents.
- Develop benchmarks that correlate with real-world production performance.
- Identify and address evaluation failure modes, including reward hacking and benchmark gaming.
- Simulate software tools such as inboxes and CRMs with sufficient fidelity to support agent interactions at scale.
Requirements
- Mid-level or senior engineering or research experience, including at least 1 to 2 years at a fast-growing or AI-focused startup.
- Hands-on AI research or engineering experience with synthetic data pipelines, benchmarks, evaluations, or post-training.
- Experience building environments or benchmarks used to train or evaluate agents and implementing agent graders or evaluation systems.
- Experience simulating complex software systems for agent tool use, identifying flaws in established benchmarks, and detecting and fixing reward hacking.
- Background in AI evaluation, benchmark design, simulation, agent systems, or related areas.
- Startup founding or accelerator experience and high-level competition experience in mathematics, programming, debate, or games are valued.
- Ability to collaborate, adapt, and work effectively in a fast-moving environment.
Benefits
- Base salary of $200,000 to $300,000 annually, with up to $350,000 for exceptional candidates.
- Additional commission-style bonus compensation of up to $200,000 or more tied to the value of delivered environments.
- Meaningful founding equity is included.
- Visa sponsorship is available.
- On-site five days a week in San Francisco, California.
