15 hours ago
Base Salary
$265k - $331k/yr
Responsibilities
- Design and deploy production AI agents using large language models, reasoning, retrieval, memory, and tool use.
- Architect systems combining LLMs, traditional machine learning, structured knowledge, enterprise data, and deterministic software.
- Build retrieval pipelines, memory systems, knowledge representations, multi-agent systems, and customer intelligence layers.
- Design evaluation frameworks using benchmarks, A/B experiments, golden datasets, regression suites, LLM-as-a-Judge, and human evaluation.
- Run experiments and ablation studies across models, prompts, retrieval strategies, reasoning techniques, memory systems, and agent architectures.
- Build reliable production AI systems with guardrails, fallback strategies, tracing, monitoring, observability, and human-in-the-loop workflows.
- Partner directly with enterprise customers to translate business challenges into AI architectures and scalable production solutions.
- Collaborate with infrastructure engineers to deploy AI systems securely in enterprise cloud environments.
Requirements
- 8+ years of software engineering, machine learning, or applied AI experience.
- Strong Python programming skills.
- Experience building production AI systems using large language models.
- Experience with modern AI tooling, including OpenAI, Claude, MCP, agent frameworks, vector databases, or retrieval systems.
- Strong understanding of machine learning fundamentals and modern language models.
- Experience designing or evaluating AI systems using quantitative metrics.
- Excellent communication skills and ability to work directly with enterprise customers.
- Preferred qualifications include experience building production AI agents or autonomous systems and understanding reasoning, retrieval, memory, planning, and tool use.
- Preferred qualifications include experience with LLM and agent evaluation frameworks, RAG, semantic search, knowledge graphs, structured knowledge, fine-tuning, distillation, reinforcement learning, small language models, model optimization, multimodal AI, and frontier foundation models.
- Preferred qualifications include experience building distributed production systems, using AWS, Azure, or GCP, Docker, Kubernetes, and production observability, and integrating AI into enterprise software.
- Preferred qualifications include experience leading technical workshops, architecture reviews, or customer design sessions.
Benefits
- Base salary, equity, and benefits are available for eligible roles.
- Comprehensive health, dental, and vision coverage.
- Retirement benefits, learning and development stipend, and generous paid time off.
- Potential commuter stipend for eligible roles.
- Full-time position located in San Francisco, New York, or Seattle; the posting directs applicants to the subtitle for the specific location.
- The posting states that the role may develop hands-on experience integrating AI systems into enterprise environments and working with customer engineering teams.
Categories
About Scale AI
Scale AI builds data annotation services and AI development tools for enterprises and government agencies, sold as a platform and managed services. Its products include the Scale Generative AI Platform for building and evaluating agents and the Data Engine for collecting, curating, and labeling training data, including RLHF and model evaluation. Founded in 2016 and headquartered in San Francisco, the company is privately held and works across domains from computer vision to LLM applications.