2 months ago
Base Salary
$216k - $270k/yr
Responsibilities
- Design and deploy production AI agents using large language models, reasoning, retrieval, memory, and tool use.
- Architect systems combining LLMs, traditional machine learning, structured knowledge, enterprise data, and deterministic software.
- Build retrieval pipelines, memory systems, knowledge representations, customer intelligence layers, and multi-agent systems.
- Design evaluation frameworks using offline benchmarks, online A/B experiments, golden datasets, regression suites, LLM-as-a-Judge, and human evaluation.
- Run experiments and ablation studies across models, prompts, retrieval strategies, reasoning techniques, memory systems, and agent architectures.
- Build reliable production AI systems with guardrails, fallback strategies, tracing, monitoring, safety, latency, and cost controls.
- Collaborate with infrastructure engineers to deploy AI systems securely in enterprise cloud environments.
- Build human-in-the-loop workflows and continuously improve agents using real-world feedback.
- Partner with enterprise customers to understand their business, data, and operational challenges.
- Translate ambiguous customer problems into production AI architectures, prototype solutions, validate them with customers, and scale reusable capabilities.
Requirements
- At least 5 years of software engineering, machine learning, or applied AI experience.
- Strong Python programming skills.
- Experience building production AI systems using LLMs.
- Experience with modern AI tooling such as OpenAI, Claude, MCP, agent frameworks, vector databases, or retrieval systems.
- Strong understanding of machine learning fundamentals and modern language models.
- Experience designing or evaluating AI systems using quantitative metrics.
- Excellent communication skills and ability to work directly with enterprise customers.
- Experience building production AI agents or autonomous systems is preferred.
- Experience with reasoning, retrieval, memory, planning, tool use, RAG, semantic search, knowledge graphs, customer intelligence systems, or structured knowledge representations is preferred.
- Experience with LLM evaluation frameworks, fine-tuning, distillation, reinforcement learning, small language models, model optimization, multimodal AI, or frontier foundation models is preferred.
- Experience building distributed production systems and integrating AI systems into enterprise software environments is preferred.
- Experience with AWS, Azure, or GCP is preferred.
- Experience with Docker, Kubernetes, CI/CD, and production observability is preferred.
- Experience leading technical workshops, architecture reviews, or customer design sessions is preferred.
Benefits
- Comprehensive health, dental, and vision coverage.
- Retirement benefits.
- Learning and development stipend.
- Generous paid time off (PTO).
- Potential commuter stipend.
- Full-time position located in San Francisco, New York, or Seattle, as referenced by the posting subtitle.
- Applicants must wait 90 days before being reconsidered for the same role.
Categories
AI ApplicationsForward Deployed
About Scale AI
Scale AI builds data annotation services and AI development tools for enterprises and government agencies, sold as a platform and managed services. Its products include the Scale Generative AI Platform for building and evaluating agents and the Data Engine for collecting, curating, and labeling training data, including RLHF and model evaluation. Founded in 2016 and headquartered in San Francisco, the company is privately held and works across domains from computer vision to LLM applications.