8 hours ago
Base Salary
$207k - $300k/yr
Responsibilities
- Define the long-term technical roadmap and architectural standards for automated evaluation infrastructure.
- Establish company-wide metric definitions and taxonomies capturing user behavior and the multidimensional quality of GenAI experiences.
- Lead the design and implementation of highly scalable, production-grade autorater systems.
- Develop statistical methodologies to validate autorater accuracy against production user signals and labeled ground truth.
- Partner with cross-functional teams to integrate statistically significant evaluations into systems.
- Mentor engineers in measurement best practices and drive technical alignment across teams.
Requirements
- Bachelor's degree or equivalent practical experience.
- 8 years of experience in software development.
- 5 years of experience testing and launching software products and 3 years of experience with software design and architecture.
- 5 years of experience leading ML design and optimizing ML infrastructure, including model deployment, model evaluation, data processing, debugging, and fine-tuning.
- 2 years of experience with state-of-the-art GenAI techniques or GenAI-related concepts such as language modeling and computer vision.
- Preferred qualifications include a Master's degree or PhD in a quantitative field such as Computer Science, Applied Mathematics, Statistics, Data Science, or Machine Learning.
- Preferred experience includes architecting and operating large-scale distributed systems or ML infrastructure in Python, C++, or Java; statistical modeling; hypothesis testing; sampling theory; reliability metrics; and tools such as Pandas and PyTorch.
- Experience leading major engineering initiatives to production deployment and driving technical alignment across teams is preferred.
Benefits
- 20% bonus target, equity, and benefits are offered in addition to base pay.
Categories
About Google
Google builds consumer and enterprise software and services including Search, Android, YouTube, Chrome, Maps, Gmail, and Google Cloud. Its business model centers on digital advertising and paid cloud, software, and hardware offerings (e.g., Pixel and Nest) for consumers, developers, and organizations. Founded in 1998 and headquartered in Mountain View, California, Google operates globally as a subsidiary of Alphabet Inc.