2 days ago
Base Salary
$160k - $250k/yr
Responsibilities
- Own agent architecture covering tool calling, state management, error recovery, model routing, and context management.
- Define task-success metrics and track workflow completion, cost, and failure modes.
- Build rigorous, reproducible evaluation and benchmarking infrastructure for agent task completion.
- Map real user workflows with researchers and domain experts and prioritize valuable agent capabilities.
- Set per-task token budgets and measure cost per completed workflow.
- Lead a small AI team through technical direction, design reviews, and code reviews while contributing hands-on production code.
- Collaborate with product, integrations, and customers to align agent behavior with real-world workflows.
Requirements
- At least 7 years of software engineering experience, including 2 or more years building LLM agents that take real-world actions.
- Deep knowledge of LLM application architecture, including model selection, context management, retrieval, tool calling, and orchestration.
- Experience creating agent evaluation and benchmarking frameworks for task completion, cost efficiency, and failure analysis.
- Strong Python skills and familiarity with function calling, tool APIs, observability or tracing, and evaluation tools.
- Hands-on technical leadership experience guiding a small engineering team while continuing to write production code.
- Experience shipping AI or LLM tools over proprietary engineering data or desktop engineering software.
- Experience with desktop automation or application APIs such as COM and familiarity with enterprise workstation constraints.
- Background in mechanical engineering software, CAD/CAE, PLM, or a related engineering domain.
Benefits
- On-site role in San Francisco, California.
- Visa sponsorship is not available.