3 hours ago
Base Salary
$209k - $429k/yr
Responsibilities
- Identify modernization opportunities and scale the roadmap for a company-wide AI/ML evaluation system.
- Define cross-team technical strategy, align partner teams, and drive adoption of shared evaluation capabilities.
- Make architectural decisions that improve the user experience and reliability of the AI/ML evaluation platform.
- Establish evaluation methodologies balancing automated metrics, expert judgment, human feedback, safety, and business outcomes.
- Lead complex cross-team workstreams, resolve technical tradeoffs and risks, and deliver impact.
Requirements
- At least 3 years of deep, hands-on experience designing and operating evaluation systems for GenAI and complex ML systems, including benchmark design, golden datasets, automated metrics, human evaluation, model-based grading, and failure analysis.
- At least 5 years of experience building scalable platforms for distributed experimentation, data and model versioning, orchestration, observability, release management, and production monitoring.
- Strong Python programming skills and a solid understanding of distributed systems principles.
- Familiarity with GenAI and RecSys evaluation libraries and frameworks, including AutoEvals, OpenEvals, and Ragas.
- Strong judgment regarding GenAI risks such as robustness, bias, privacy, security, and limitations of automated evaluation signals.
- Bachelor’s degree in Computer Science, Engineering, or a related field, or equivalent experience.
Benefits
- US-based remote role with in-person collaboration required 1–2 times per quarter.
- Eligible for equity in addition to base salary.
- Not eligible for relocation assistance.
Tech Stack
Categories
About Pinterest
Pinterest builds a visual discovery and bookmarking platform where people search, save, and shop ideas across home, fashion, food, and more. The company monetizes primarily through advertising and shopping integrations, using visual search and recommendation systems to connect users and merchants. Founded in 2010 and headquartered in San Francisco, Pinterest is a public company on the NYSE (PINS) with around 482 million monthly users worldwide.