21 hours ago
San Francisco, CA, USAEntry Level / Mid Level
Base Salary
$200k - $350k/yr
Responsibilities
- Design synthetic company environments with realistic email, records, and files for AI agents.
- Build robust graders and evaluation systems for advanced AI agents.
- Develop benchmarks correlated with real-world production performance.
- Identify and resolve evaluation failures, reward hacking, and benchmark gaming.
- Simulate software systems such as inboxes and CRMs that can withstand agent tool calls at scale.
Requirements
- Hands-on experience with synthetic data pipelines, benchmarks, evaluations, or post-training.
- Experience building environments or benchmarks used to train or evaluate agents at advanced AI labs.
- Experience implementing graders or evaluation systems.
- At least 1 to 2 years at a fast-growing or AI-focused startup, including experience building synthetic environments or production-like simulations.
- Ability to identify flaws in established benchmarks and detect and fix reward hacking in evaluation systems.
- Strong academic or equivalent credentials, such as a top-tier university background, founding experience, or participation in a leading startup program.
- High-level competition experience in mathematics, programming, debate, or games is valued.
Benefits
- $200,000 to $300,000 annual base salary, with up to $350,000 for exceptional candidates.
- Additional commission-style bonus of up to $200,000 or more tied to the value of delivered environments.
- Meaningful founding equity.
- On-site five days a week in San Francisco, California.
- Visa sponsorship is not available.