AI Engineer, Evals & Agent Quality
Town.com, Inc.7 days ago
San Francisco, CA, USASenior / Staff+
Base Salary
$250k - $300k/yr
Responsibilities
- Build a generalized evaluation system for assistant quality across product surfaces and multi-step agent trajectories.
- Create and maintain golden datasets and labeling loops to validate improvements and prevent regressions.
- Build model-routing and online-evaluation tooling to balance model cost, quality, and speed.
- Make prompts and system changes measurable and partner with engineers to instrument quality and close the loop from signals to fixes.
Requirements
- Experience building or owning LLM evaluation systems or offline/online quality measurement at scale.
- Hands-on familiarity with evaluation tooling and frameworks and strong judgment about measurement approaches.
- Ability to reason about model routing and tradeoffs between models.
- Ability to ship quality improvements rather than only dashboards and metrics.
- Senior or staff-level engineering experience in greenfield environments.
Benefits
- Full-time role based in San Francisco, California.
- Five days per week working in person from the Financial District office.
Categories