8 months ago
San Francisco, CA, USA or New York, NY, USASenior / Staff+
H1B sponsor
Base Salary
$218k - $273k/yr
Responsibilities
- Design and implement end-to-end agent systems combining LLM reasoning, tool use, memory, and control logic
- Build scalable and reliable agent architectures deployable across customers with varying data, tools, and constraints
- Develop evaluation frameworks, datasets, environments, and metrics for production agent performance and business impact
- Translate enterprise requirements into robust agent designs with product managers, customers, data annotators, and engineering teams
- Productionize planning, multi-step reasoning, tool use, and multi-agent techniques into maintainable and observable systems
- Own deployment, monitoring, failure analysis, and continuous improvement of agent systems
- Contribute to technical direction, architecture, and best practices for general agent development, with increasing Staff-level leadership scope
Requirements
- 5+ years of experience building and deploying machine learning or AI systems for real-world production use cases
- Bachelor’s and/or master’s degree in Computer Science, Machine Learning, AI, or equivalent practical experience
- Deep understanding of modern LLMs, prompt-, context-, and system-level optimization, and agentic system design
- Proficiency in Python and production-quality, testable, maintainable software development
- Experience integrating models with external tools, APIs, databases, and services
- Ability to balance research-driven approaches with pragmatic product constraints
- Strong communication skills and comfort working in customer-facing or cross-functional environments
- Preferred experience with OpenAI APIs, commercial or open-source LLMs, agent frameworks, orchestration layers, and workflow systems
- Preferred familiarity with evaluation, monitoring, and observability for LLM-powered systems
- Preferred experience deploying machine learning systems in cloud environments and operating them at scale
- Preferred experience fine-tuning or adapting foundation models with SFT, RLVR, and LoRA
Benefits
- Comprehensive health, dental, and vision coverage
- Retirement benefits
- Learning and development stipend
- Generous paid time off
- Potential commuter stipend
- Full-time position located in San Francisco, New York, or Seattle
- 90-day waiting period before reconsideration for the same role
Tech Stack
Categories
About Scale AI
Scale AI builds data annotation services and AI development tools for enterprises and government agencies, sold as a platform and managed services. Its products include the Scale Generative AI Platform for building and evaluating agents and the Data Engine for collecting, curating, and labeling training data, including RLHF and model evaluation. Founded in 2016 and headquartered in San Francisco, the company is privately held and works across domains from computer vision to LLM applications.