2 days ago
Base Salary
$160k - $250k/yr
Responsibilities
- Lead the design and development of agent capabilities for multi-step workflows in desktop engineering software.
- Define evaluation frameworks and reproducible benchmarks based on validated user stories, tracking task completion and failure modes.
- Set workflow token budgets and measure cost per completed task.
- Translate engineering workflows into validated test cases with user researchers and domain experts.
- Prioritize workflow coverage based on customer value and technical feasibility.
- Own architecture decisions involving tool calling, state management, error recovery, model routing, and context management.
- Provide technical direction, review designs and code, unblock teammates, and contribute production software.
- Collaborate with product, integrations, and customers to align agent behavior with real-world needs.
Requirements
- At least 7 years of software engineering experience, including 2 or more years building LLM-based agents that take real-world actions.
- Deep experience with LLM application architecture, including model selection, context management, retrieval, tool calling, and orchestration.
- Strong Python skills and familiarity with function calling, tool APIs, tracing or observability, and evaluation tools.
- Experience building agent evaluation and benchmarking systems that measure task completion, cost efficiency, and failure modes.
- Hands-on technical leadership experience setting direction and reviewing code for a small team while continuing to ship production software.
- Experience applying AI or LLM tooling to proprietary engineering data or desktop engineering software.
- Background in mechanical engineering, CAD, CAE, PLM, or an adjacent domain is valuable.
- Familiarity with desktop automation or programmatic application control such as COM, including enterprise workstation constraints.
Benefits
- On-site role in San Francisco, California.
- Visa sponsorship is not available.
