7 days ago
Responsibilities
- Develop scalable quality-control systems and methodologies for AI and human-in-the-loop evaluation.
- Design ground-truth generation pipelines across varied task types, annotation modalities, and cold-start scenarios.
- Build real-time validation, monitoring, anomaly detection, and drift detection systems for live data and ML pipelines.
- Perform generative-AI evaluation, including LLM-as-a-judge design, meta-evaluation, failure-mode analysis, and calibration or reference-guided grading.
- Prototype, validate, deploy, and monitor production LLM-based pipelines and agents.
- Analyze disagreement root causes and convert findings into corrective actions.
- Collaborate with downstream users and cross-functional stakeholders to incorporate feedback into system design and communicate findings clearly.
- Design configurable and extensible tooling that can be used by practitioners who did not build it.
Requirements
- At least 5 years of industry experience in applied science or machine learning.
- Production experience building or operating evaluation, annotation, or quality-assurance pipelines.
- Hands-on experience designing ground-truth generation pipelines across varied tasks and annotation modalities.
- Experience building real-time monitoring or anomaly/drift detection systems for live data or ML pipelines.
- Working knowledge of generative-AI evaluation methodologies, including LLM-as-a-judge, meta-evaluation, failure-mode analysis, and calibration or reference-guided grading.
- Strong software engineering fundamentals and proficiency in Python and relevant ML frameworks.
- Production experience building, deploying, and monitoring LLM-based pipelines and agents.
- Ability to work with stakeholders, incorporate feedback into system design, and communicate with technical and non-technical audiences.
- MS or PhD in Computer Science, Machine Learning, Statistics, or a related quantitative field, or equivalent practical experience.
- Preferred qualifications include a PhD, experience building configurable and extensible practitioner tooling, strong communication and influencing skills, and interest in using AI to improve efficiency and scale.
Tech Stack
Categories
About Apple
We’re a diverse collective of thinkers and doers, continually reimagining what’s possible to help us all do what we love in new ways. And the same innovation that goes into our products also applies to our practices — strengthening our commitment to leave the world better than we found it. This is where your work can make a difference in people’s lives. Including your own. Apple is an equal opportunity employer that is committed to inclusion and diversity. Visit apple.com/careers to learn more.