5 hours ago
San Francisco, CA, USAEntry Level / Mid Level / Senior
Base Salary
$120k - $200k/yr
Responsibilities
- Detect failures in production AI agents, including omissions, incorrect responses, and patterns across large volumes of traces.
- Use user behavior signals such as rephrasing, abandonment, and retries to identify potential failures.
- Build evaluation methods for detection quality, including evaluation without ground-truth labels.
- Verify that proposed fixes reproduce and resolve failures and avoid recommendations when evidence is weak.
- Improve detection cost efficiency by selecting appropriate models and methods for each task.
- Review production traces and guide product improvements.
Requirements
- Demonstrated AI/ML engineering experience; candidates with 0 to 10 years of experience are considered.
- Experience shipping products used by people and working as an individual-contributor engineer.
- Hands-on production experience with LLMs, evaluations, LLM-as-judge, embeddings, and selecting smaller models or no model when appropriate.
- Experience running AI agents in production and handling their failures.
- Strong research judgment and the ability to distinguish meaningful improvements from noise when direct ground truth is unavailable.
Benefits
- Base salary range of $120,000 to $200,000 annually.
- On-site work in San Francisco, United States.
- Visa sponsorship is not available; the role requires authorization to work in the United States without visa sponsorship.