3 months ago
Base Salary
$216k - $270k/yr
Responsibilities
- Build production observability and instrumentation for agent behavior.
- Design evaluation methodologies and metrics and automate them at scale across customer use cases.
- Build, deploy, and own ML systems that detect drift, anomalies, or misalignment in agent behavior.
- Design and run rigorous experiments to validate model and agent improvements before release.
- Collaborate with software engineers on platform infrastructure while taking work from concept through production.
- Work with product, customers, data annotators, Forward Deployed Engineers, and other engineering teams to develop robust platform capabilities.
- Contribute to novel methods for agent evaluation and improvement or to reliable production ML systems at scale.
Requirements
- At least 5 years of experience as a machine learning engineer or applied scientist, ideally working on production ML or LLM-powered systems.
- Strong grounding in at least two of evaluation, monitoring, or continuous-learning infrastructure; agent system design; or new methods, reward models, and model training or fine-tuning.
- Hands-on experience with LLMs and agent architectures, including tool use, planning, and multi-agent orchestration.
- Experience partnering with software engineers to productionize research and experimental work.
- Strong experimentation skills, including hypothesis development, statistical grounding, and rigorous validation.
- Experience collaborating across functions to turn ambiguous enterprise or government requirements into production capabilities.
- Ability to provide and receive substantive design and code feedback and mentor others.
- Preferred experience includes RLHF, SFT, fine-tuning or reinforcement-learning workflows, reward modeling, verifiable-reward systems, model or systems optimization, research or open-source contributions, regulated or enterprise environments, and senior/staff-level technical review or mentoring.
Benefits
- Comprehensive health, dental, and vision coverage.
- Retirement benefits, learning and development stipend, and generous paid time off.
- Potential commuter stipend and equity eligibility for eligible roles.
- Full-time position located in San Francisco, New York, or Seattle.
- 90-day waiting period before reconsideration for the same role.
Categories
About Scale AI
Scale AI builds data annotation services and AI development tools for enterprises and government agencies, sold as a platform and managed services. Its products include the Scale Generative AI Platform for building and evaluating agents and the Data Engine for collecting, curating, and labeling training data, including RLHF and model evaluation. Founded in 2016 and headquartered in San Francisco, the company is privately held and works across domains from computer vision to LLM applications.