AI Engineer, Evals & Agent Quality
Town.com, Inc.2 months ago
San Francisco, CA, USASenior / Staff+
Base Salary
$250k - $300k/yr
Responsibilities
- Build a generalized evaluation system for assistant quality across product surfaces and multi-step agent trajectories.
- Create and maintain golden datasets and labeling loops to validate improvements and prevent regressions.
- Build model-routing and online-evaluation tooling to balance model cost, quality, and speed.
- Make prompts and system changes measurable and partner with engineers to instrument quality and close the loop from signals to fixes.
Requirements
- Experience building or owning LLM evaluation systems or offline/online quality measurement at scale.
- Hands-on familiarity with evaluation tooling and frameworks and strong judgment about measurement approaches.
- Ability to reason about model routing and tradeoffs between models.
- Ability to ship quality improvements rather than only dashboards and metrics.
- Senior or staff-level engineering experience in greenfield environments.
Benefits
- Full-time role based in San Francisco, California.
- Five days per week working in person from the Financial District office.
Categories
About Town.com, Inc.
Town.com, Inc. builds an AI-powered assistant that connects to users’ existing tools to triage email, draft messages in their voice, manage calendars, and run cross-app workflows. The product targets professionals and teams seeking an agent that learns over time and performs work on their behalf. Founded in 2024 and headquartered in San Francisco, the company is privately held.