5 hours ago
San Francisco, CA, USAEntry Level / Mid Level / Senior
Base Salary
$120k - $200k/yr
Responsibilities
- Detect subtle failures in production agents, including incorrect compliant responses, omissions, and patterns across traces.
- Use rephrasing, abandonment, retries, and other indirect user signals as evidence of agent failures.
- Build evaluations to measure detection quality and improve performance without labeled ground truth.
- Reproduce failures, verify proposed fixes, and prevent low-confidence patch generation.
- Optimize the cost and quality tradeoffs of model-based evaluation, including when to use smaller models or no model.
- Review production traces to identify problems and guide system improvements.
Requirements
- Experience may range from new graduate to approximately 10 years.
- Track record of shipping products or systems used by real people.
- Hands-on production experience with LLMs, evaluations, LLM-as-judge approaches, and embeddings.
- Strong research judgment and ability to distinguish meaningful improvements from noise when ground truth is limited.
- Experience operating agents in production and learning from their failures is a plus.
Benefits
- Salary range of $120,000 to $200,000 USD annually.
- On-site role based in San Francisco, United States.