1 day ago
Vancouver, CanadaSenior
Responsibilities
- Define evaluation strategies and metrics for ML models, LLMs, and AI features based on product use cases.
- Create, curate, and improve representative evaluation datasets and dataset-quality workflows.
- Design, build, maintain, and optimize scalable evaluation infrastructure with reproducible runs, tracked baselines, and model-version comparisons.
- Apply ML, LLMs, and agentic workflows to automated judging, test generation, and evaluation analysis.
- Drive end-to-end AI feature evaluation and quality from model checkpoints through integrated shipping features, including issue triage and resolution.
- Translate evaluation results into ship/no-ship risk assessments and recommendations for technical and non-technical audiences.
- Partner with and drive cross-functional machine learning, infrastructure, product, and design teams.
Requirements
- At least 5 years of industry experience in machine learning or software engineering focused on ML/AI model evaluation and quality.
- Experience with evaluation methodology for ML models, LLMs, and AI products or features.
- Strong understanding of the ML development cycle, including data, training, evaluation, deployment, and monitoring.
- Solid machine learning fundamentals and LLM knowledge, including agentic workflows.
- Experience applying LLMs or agentic workflows to ML evaluation and quality.
- Experience with data curation, dataset creation, dataset quality, and iterative dataset improvement workflows.
- Experience designing, implementing, maintaining, or optimizing ML evaluation infrastructure at scale.
- Experience designing evaluation metrics for models, products, and use cases, and driving end-to-end AI product evaluation and quality.
- Fluency in quality reporting and presenting quality data to technical and non-technical audiences.
- Strong communication skills and experience working with and driving cross-functional partners.
- Proficiency in Python and/or Swift or comparable languages.
- Bachelor's degree in Computer Science, Data Science, Machine Learning, or a related field, or equivalent experience.
- Preferred: knowledge of software development life cycle and software quality assurance methodologies.
- Preferred: experience with objective image and video quality evaluation, color science, or subjective image/video quality review.
- Preferred: passion for creative applications or experience as a photo, video, music, or design creator.
- Preferred: familiarity with iOS and macOS.
Tech Stack
Categories
About Apple
Apple designs and sells consumer electronics, software, and services for consumers and professionals worldwide, including iPhone, Mac, iPad, Apple Watch, and AirPods, plus platforms like iOS/macOS and services such as the App Store, iCloud, Music, and TV+. Its business combines device sales with services and subscriptions and in-house silicon design. Founded in 1976, Apple is headquartered in Cupertino, California, and trades on NASDAQ as AAPL.