about 2 hours ago
Base Salary
$230k - $385k/yr
Responsibilities
- Own the production improvement loop across agent behavior and business outcomes.
- Instrument agent workflows to understand model interactions and outcomes.
- Define quality standards and production monitoring for GTM workflows.
- Investigate underperformance in agent behavior across various contexts.
- Design and implement targeted behavior improvements.
- Build backend services, APIs, and feedback pipelines for agent observability.
- Run experiments and production replays to measure improvements.
- Collaborate with cross-functional teams to prioritize high-value problems.
Requirements
- 4+ years of experience in software, backend, applied AI, or product engineering.
- Experience building AI agents or LLM-powered applications.
- Ability to diagnose and improve agent behavior using production data.
- Practical experience with evaluation design and controlled experiments.
- Strong backend engineering skills in Python, APIs, and data pipelines.
- Strong product judgment connecting technical changes to customer experience.
- Comfort working across various aspects of AI and human-in-the-loop decisions.
- Ability to collaborate with both technical and non-technical partners.