2 months ago
San Francisco, CA, USAMid Level
Base Salary
$250k - $290k/yr
Responsibilities
- Design metrics that define quality across diverse websites, formats, and edge cases.
- Build scalable evaluation pipelines and harnesses to measure output quality rigorously.
- Generate and curate trustworthy evaluation datasets.
- Own the feedback loop from output quality to model and product decisions.
- Translate evaluation results into actionable answers for the broader team.
Requirements
- 4+ years of experience in ML, research engineering, or data-heavy backend engineering.
- Demonstrated experience building or conducting real evaluation work.
- Ability to build evaluation systems rather than only run existing benchmarks.
- Ability to define rigorous metrics for ambiguous quality problems.
- Ability to combine engineering implementation with research methodology.
- Comfort working independently, moving quickly, and iterating based on measurement.
- Must be legally authorized to work in the United States; visa sponsorship is not currently available.
Benefits
- Competitive equity.
- 15 days of mandatory PTO, with additional time available after 24 days; holidays are excluded.
- 12 weeks of fully paid parental leave for all parents.
- $100/month wellness stipend.
- Up to $1,000/year for learning and development.
- Team offsites.
- Three paid months of sabbatical after four years.
- For U.S.-based full-time employees: medical, dental, and vision coverage, employer-paid short- and long-term disability and life insurance, optional supplemental insurance, telehealth, 401(k), FSA and commuter benefits, and pet insurance.
- For San Francisco-based employees: snacks, drinks, team lunches, and a loaner electric bike.
- Full-time, onsite role in San Francisco.
- Interview includes a paid 1–2 week work trial at a contractor rate; the trial is remote-friendly and can accommodate current commitments.