Base Salary
$216k - $270k/yr
Responsibilities
- Build production observability and instrumentation for agent behavior.
- Design evaluation methodologies and metrics and automate them at scale across customer use cases.
- Build, deploy, and own ML systems that detect drift, anomalies, or misalignment in agent behavior.
- Design and run rigorous experiments to validate model and agent improvements before release.
- Collaborate with software engineers on platform infrastructure while taking work from concept through production.
- Work with product, customers, data annotators, Forward Deployed Engineers, and other engineering teams to develop robust platform capabilities.
- Contribute to novel methods for agent evaluation and improvement or to reliable production ML systems at scale.
Requirements
- At least 5 years of experience as a machine learning engineer or applied scientist, ideally working on production ML or LLM-powered systems.
- Strong grounding in at least two of evaluation, monitoring, or continuous-learning infrastructure; agent system design; or new methods, reward models, and model training or fine-tuning.
- Hands-on experience with LLMs and agent architectures, including tool use, planning, and multi-agent orchestration.
- Experience partnering with software engineers to productionize research and experimental work.
- Strong experimentation skills, including hypothesis development, statistical grounding, and rigorous validation.
- Experience collaborating across functions to turn ambiguous enterprise or government requirements into production capabilities.
- Ability to provide and receive substantive design and code feedback and mentor others.
- Preferred experience includes RLHF, SFT, fine-tuning or reinforcement-learning workflows, reward modeling, verifiable-reward systems, model or systems optimization, research or open-source contributions, regulated or enterprise environments, and senior/staff-level technical review or mentoring.
Benefits
- Comprehensive health, dental, and vision coverage.
- Retirement benefits, learning and development stipend, and generous paid time off.
- Potential commuter stipend and equity eligibility for eligible roles.
- Full-time position located in San Francisco, New York, or Seattle.
- 90-day waiting period before reconsideration for the same role.
Categories
About Scale AI
Scale’s mission is to develop reliable AI systems for the world’s most important decisions. We provide the high-quality data and full-stack technologies that power the world’s leading models, and help enterprises and governments build, deploy, and oversee AI applications that deliver real impact. The Scale Generative AI Platform allows customers to build, evaluate, and control advanced AI agents and applications that continuously improve. The Scale Data Engine provides the technology to collect, curate, and annotate high-quality datasets. Through our Scale Labs, we test models with rigorous benchmarks and novel research to ensure breakthroughs translate into systems people can trust. Scale powers the most advanced LLMs and generative models in the world through RLHF, data generation and model evaluation. We work with industry leaders like Meta, Cisco, DLA Piper, Mayo Clinic, Time Inc., the Government of Qatar, and U.S. government agencies including the Army and Air Force.