4 hours ago
San Francisco, CA, USAEntry Level / Mid Level / Senior
Base Salary
$120k - $200k/yr
Responsibilities
- Detect subtle failures and failure patterns across large volumes of production agent traces.
- Use rephrasing, abandonment, retries, and other user behavior signals as evidence of agent failures.
- Build evaluation methods to measure detection precision when labeled ground truth is limited or unavailable.
- Reproduce failures, verify proposed fixes, and prevent low-confidence patches from being applied.
- Improve detection cost efficiency by selecting appropriate models and approaches for each event.
- Review production customer traces to identify problems and improvement opportunities.
Requirements
- Demonstrated AI/ML engineering experience; candidates with 0 to 10 years of experience are welcome, including new graduates and people without a traditional degree.
- Hands-on production experience with LLMs, evaluations, LLM-as-judge, embeddings, and deciding when to use a smaller model or no model.
- Individual-contributor engineering background and a track record of shipping products used by people.
- Strong research judgment and ability to distinguish meaningful improvements from noise when ground truth is unavailable.
- Experience running agents in production and learning from their failures is preferred.
Benefits
- Base salary of $120,000 to $200,000 annually.
- On-site work in San Francisco, United States.
- Visa sponsorship is not available.