8 months ago
San Francisco, CA, USAMid Level
Base Salary
$180k - $300k/yr
Responsibilities
- Build proprietary benchmarks and datasets for fraud, identity, and risk workflows.
- Design and run offline and online evaluations against real customer tasks.
- Define quality metrics including precision, calibration, consistency, abstention, and failure handling.
- Analyze model and agent failures and translate them into improved evaluations, datasets, and training loops.
- Build reusable evaluation tools and quality components across product workflows.
- Partner with research, engineering, product, and design teams on rigorous experimentation.
- Advance evaluation practices for regulated, adversarial, and high-consequence AI systems.
Requirements
- Experience training, evaluating, or improving modern ML systems.
- Strong programming skills and comfort with research-heavy codebases.
- Experience building benchmarks, datasets, evaluation pipelines, or quality systems.
- Familiarity with LLMs, agent systems, retrieval, post-training, or adjacent areas.
- Ability to design clean experiments and draw reliable conclusions from noisy results.
- Strong engineering judgment and a bias toward building.
- Interest in fraud, risk, trust and safety, compliance, or other regulated and adversarial domains.
Benefits
- Competitive salary and meaningful equity.
- Platinum-level medical, dental, and vision insurance.
- Unlimited PTO, sick leave, and parental leave.
- Up to $100 per month reimbursement for personal health and wellness expenses.
- 401(k) plan.
- Fully in-person work in San Francisco.