4 hours ago
San Francisco, CA, USAMid Level
Base Salary
$200k - $300k/yr
Responsibilities
- Build evaluation systems using real conversations and user preferences to measure product quality.
- Post-train large language models with supervised fine-tuning, preference tuning, and distillation, and ship improvements to production.
- Develop and evaluate model routing across smaller and more capable models.
- Balance response quality, latency, and model costs across the product.
- Improve agent reliability for website navigation, sign-in, and checkout flows without APIs.
- Test adversarial inputs, data exposure, and unintended actions, and build safeguards and evaluations.
- Apply ML to memory, retrieval, tool selection, and decisions about when to act or ask the user.
- Review real conversations, identify improvement opportunities, and help set research direction as the team grows.
Requirements
- At least 4 years of ML or software engineering experience in an AI lab, AI product company, or equivalent environment.
- Hands-on experience fine-tuning or post-training large language models and shipping results to production.
- Experience designing evaluations such as LLM judges, human preference data, or traffic replay for product decisions.
- Background in one or more of red teaming, browser or computer-use agents, model routing, or production agents with tool use and long-running memory.
- Strong Python skills and familiarity with the ML stack.
- Comfort using TypeScript for product code.
- Ability and interest to take ownership and set research direction with a founding-team mindset.
Benefits
- Annual salary range of $200,000 to $300,000.
- Full-time, on-site work in San Francisco, California.