15 days ago
Responsibilities
- Define evaluation strategies and metrics for ML models, LLMs, and AI features based on product use cases.
- Create, curate, measure, and improve evaluation datasets tailored to individual products.
- Design, build, maintain, and optimize scalable evaluation infrastructure with reproducible runs, tracked baselines, and model-version comparisons.
- Apply ML, LLMs, and agentic workflows to automated judging, test generation, and evaluation analysis.
- Drive end-to-end AI feature evaluation from model checkpoints through integrated shipping features, including issue triage and resolution.
- Report evaluation quality and provide ship/no-ship risk assessments and recommendations.
- Partner with and drive cross-functional teams across ML engineering, infrastructure, product, and design.
Requirements
- 5+ years of industry experience in machine learning or software engineering focused on ML/AI model evaluation and quality.
- Experience with evaluation methodologies for ML models, LLMs, and AI products or features.
- Strong understanding of the ML development cycle, including data, training, evaluation, deployment, and monitoring.
- Strong ML fundamentals and LLM knowledge, including agentic workflows.
- Experience applying LLMs or agentic workflows to ML evaluation and quality.
- Experience with data curation, dataset creation, dataset quality, and iterative dataset improvement workflows.
- Experience designing, implementing, maintaining, or optimizing ML evaluation infrastructure at scale.
- Experience designing evaluation metrics and driving end-to-end quality for AI products.
- Ability to report and present quality data to technical and non-technical audiences, with strong communication and cross-functional collaboration skills.
- Proficiency in Python and/or Swift or comparable languages.
- Bachelor’s degree in Computer Science, Data Science, Machine Learning, or a related field, or equivalent experience.
- Preferred: knowledge of software development life cycle and software quality assurance methodologies.
- Preferred: experience with objective image and video quality evaluation, color science, or subjective image/video quality review.
- Preferred: passion for creative applications or experience as a photo, video, music, or design creator.
- Preferred: familiarity with iOS and macOS.
Tech Stack
Categories
About Apple
Apple designs and sells consumer electronics, software, and services for consumers and professionals worldwide, including iPhone, Mac, iPad, Apple Watch, and AirPods, plus platforms like iOS/macOS and services such as the App Store, iCloud, Music, and TV+. Its business combines device sales with services and subscriptions and in-house silicon design. Founded in 1976, Apple is headquartered in Cupertino, California, and trades on NASDAQ as AAPL.