3 days ago
San Francisco, CA, USAEntry Level / Mid Level / Senior
Base Salary
$120k - $200k/yr
Responsibilities
- Detect subtle failures and issue patterns across large volumes of production AI-agent traces.
- Use rephrasing, abandonment, retries, and other user-behavior signals to identify unreported issues.
- Develop evaluation methods for measuring detection quality without labeled ground truth.
- Build patch-generation workflows that reproduce failures, verify fixes, and avoid unreliable changes.
- Select appropriate models and methods to improve the cost-effectiveness of failure detection.
- Regularly review production customer traces to identify improvement opportunities.
Requirements
- Applicants with 0 to 10 years of experience are welcome, including new graduates and candidates without a degree.
- Experience shipping products or systems used by people.
- Hands-on experience with production LLMs, evaluations, LLM-as-judge approaches, and embeddings.
- Strong judgment in assessing research results and distinguishing meaningful improvements from noise when ground truth is unavailable.
- Experience operating AI agents in production is preferred.
- Must be authorized to work in the United States without visa sponsorship.
Benefits
- Annual base salary range of $120,000 to $200,000 USD.
- On-site work in San Francisco, United States.