2 months ago
Responsibilities
- Develop scalable quality-control systems and methodologies for AI and human-in-the-loop evaluation.
- Design ground-truth generation pipelines across varied task types, annotation modalities, and cold-start scenarios.
- Build real-time validation, monitoring, anomaly detection, and drift detection systems for live data and ML pipelines.
- Perform generative-AI evaluation, including LLM-as-a-judge design, meta-evaluation, failure-mode analysis, and calibration or reference-guided grading.
- Prototype, validate, deploy, and monitor production LLM-based pipelines and agents.
- Analyze disagreement root causes and convert findings into corrective actions.
- Collaborate with downstream users and cross-functional stakeholders to incorporate feedback into system design and communicate findings clearly.
- Design configurable and extensible tooling that can be used by practitioners who did not build it.
Requirements
- At least 5 years of industry experience in applied science or machine learning.
- Production experience building or operating evaluation, annotation, or quality-assurance pipelines.
- Hands-on experience designing ground-truth generation pipelines across varied tasks and annotation modalities.
- Experience building real-time monitoring or anomaly/drift detection systems for live data or ML pipelines.
- Working knowledge of generative-AI evaluation methodologies, including LLM-as-a-judge, meta-evaluation, failure-mode analysis, and calibration or reference-guided grading.
- Strong software engineering fundamentals and proficiency in Python and relevant ML frameworks.
- Production experience building, deploying, and monitoring LLM-based pipelines and agents.
- Ability to work with stakeholders, incorporate feedback into system design, and communicate with technical and non-technical audiences.
- MS or PhD in Computer Science, Machine Learning, Statistics, or a related quantitative field, or equivalent practical experience.
- Preferred qualifications include a PhD, experience building configurable and extensible practitioner tooling, strong communication and influencing skills, and interest in using AI to improve efficiency and scale.
Tech Stack
Categories
About Apple
Apple designs and sells consumer electronics, software, and services for consumers and professionals worldwide, including iPhone, Mac, iPad, Apple Watch, and AirPods, plus platforms like iOS/macOS and services such as the App Store, iCloud, Music, and TV+. Its business combines device sales with services and subscriptions and in-house silicon design. Founded in 1976, Apple is headquartered in Cupertino, California, and trades on NASDAQ as AAPL.