2 months ago
Base Salary
$244k - $305k/yr
Responsibilities
- Define evaluation strategies and success metrics for generative AI systems.
- Build and scale evaluation frameworks using golden sets, synthetic data, automated regressions, rubric-based grading, and LLM-as-judge methods where appropriate.
- Design data-flywheel systems covering instrumentation, feedback collection, data quality, labeling, dataset versioning, and governance.
- Lead cross-functional quality initiatives across product, operations, and engineering.
- Develop and productionize pipelines for dataset creation, model monitoring, evaluation at scale, and continuous pre-deployment and post-deployment testing.
- Drive architecture and technical decisions for evaluation and data infrastructure while balancing speed, rigor, cost, and safety.
Requirements
- PhD in Computer Science, Mathematics, Statistics, or a related technical field, or equivalent practical experience.
- 10+ years building, testing, and shipping ML/AI systems end to end.
- 2+ years of experience with generative AI or LLM systems in production.
- 5+ years leading large, ambiguous technical initiatives as a senior individual contributor and influencing cross-team roadmaps and technical direction.
- Deep expertise in evaluation methodology, including offline/online alignment, metric design, human-in-the-loop evaluation, A/B testing, power analysis, and regression testing.
- Hands-on experience with GenAI systems, including orchestration, retrieval, tool calling, and memory.
- Experience building data pipelines and quality systems involving labeling workflows, dataset curation, versioning, monitoring, and governance.
- Solid ML fundamentals covering model selection, training and serving, monitoring, reliability, and model lifecycle management.
- Preferred: experience applying ML or AI to customer support workflows such as agent assist, classification or routing, resolution recommendation, or quality assurance.
- Preferred: experience building evaluation platforms for agent behavior validation, safety and guardrails, and continuous improvement.
- Preferred: experience taking applied-AI evaluation and data-flywheel work from incubation to production while maintaining scientific rigor.
Benefits
- US remote eligible, with occasional Airbnb office work or offsites as agreed with the manager.
- May be eligible for bonus, equity, benefits, and Employee Travel Credits.
- Employees must live in a state where Airbnb, Inc. has a registered entity.
Categories
About Airbnb
Airbnb operates a global online marketplace that connects hosts offering homes, rooms, and unique properties with travelers seeking short-term stays and experiences. The company earns revenue primarily through service fees on bookings; founded in 2007 and headquartered in San Francisco, it is a public company on NASDAQ (ABNB) with over 5 million hosts and guest arrivals recorded in nearly every country.