10 hours ago
Base Salary
$200k - $240k/yr
Responsibilities
- Own evaluation infrastructure for quickly configuring, concurrently running, automatically executing, and monitoring modeling tests.
- Implement deterministic, traceable measurement pipelines according to scientists’ specifications.
- Maintain versioned evaluation datasets, including held-out sets excluded from training.
- Build and maintain pipelines for human-participant data, external datasets, and model-training data, including participant record linkage.
- Keep research code aligned with the production product so validated improvements can ship promptly.
- Detect and prevent threats to evaluation validity.
- Track compute and API spending across experiments and assess production-stage cost implications.
- Manage repositories, code review, documentation, and reproducible coding workflows.
- Identify opportunities to improve implementation correctness and efficiency and propose changes.
Requirements
- Several years of software engineering experience, including experience supporting researchers in a lab or comparable setting.
- Experience translating scientific requirements and experiment designs into code.
- Experience building and operating large-scale data pipelines on human-behavior data and validating them against researcher-defined criteria.
- Strong Python skills, including the ability to read and audit code produced by agentic coding tools.
- Knowledge of file systems, software libraries, databases, SQL, tabular data, and Git-based workflows.
- Experience with cloud infrastructure and concurrent job execution.
- Working knowledge of machine learning, transformer-based language models, model APIs, and fine-tuning.
- Demonstrated ability to write deterministic, reproducible, cost-aware, and well-documented code.
- Preferred: experience engineering LLM systems, including prompt pipelines, evaluation harnesses, embedding computation, agentic systems, or fine-tuning infrastructure.
- Preferred: experience fine-tuning open-weight language models on multi-GPU hardware.
- Preferred: experience serving open-weight models with vLLM or SGLang, including batched inference and prefix caching.
- Preferred: experience with experiment tracking, machine-learning cost monitoring, GPU provisioning across cloud providers, survey or panel data, data collection platforms, model activations, generative agents, digital twins, or human-respondent simulation.
Categories
Data EngineeringML Engineering
About Outset
Outset builds an AI‑moderated research platform that runs qualitative user interviews and automates analysis for product, UX, and marketing teams. Delivered as a SaaS platform, it helps organizations conduct discovery, concept, brand, and pricing research at scale. Founded in 2022 and headquartered in San Francisco, the privately held company counts customers such as Microsoft, Nestlé, Uber, and HubSpot.