Senior AI Engineer, Quality
FieldguideBase Salary
$200k - $250k/yr
Responsibilities
- Design and build a unified evaluation platform as the source of truth for agentic systems and audit workflows.
- Build production observability, execution tracing, failure-mode detection, and feedback loops that turn failures into evaluation cases.
- Own evaluation infrastructure integrations with LangSmith and LangGraph.
- Integrate and orchestrate LLMs, tools, retrieval systems, and logic into reliable agent experiences.
- Build automated pipelines and evaluation harnesses to assess new models across critical workflows within hours.
- Create comparison frameworks measuring effectiveness, consistency, latency, and cost across model versions.
- Design guardrails and monitoring systems that detect quality regressions before they reach customers.
- Create evaluation datasets from curated production traces with subject-matter experts and ML engineers.
- Design prompts, retrieval pipelines, and agent orchestration systems for reliable operation at scale.
- Define evaluation standards and best practices, and help engineering teams adopt evaluation-driven development.
- Partner with product and ML engineers to integrate evaluation requirements into agent development from the beginning.
Requirements
- Multiple years of experience shipping production software in complex real-world systems are typical for strong candidates.
- Experience with TypeScript, React, Python, and Postgres.
- Experience building and deploying LLM-powered features serving production traffic.
- Experience implementing evaluation frameworks for model outputs and agent behaviors.
- Experience designing observability or tracing infrastructure for AI/ML systems.
- Experience with vector databases, embedding models, and RAG architectures.
- Experience with evaluation platforms such as LangSmith, Langfuse, or similar.
- Ability to operate in ambiguity, take responsibility for outcomes, and own large product areas.
- Strong product judgment, data-driven rigor, and a production-oriented approach.
- Experience with audit and accounting workflows is not required.
Benefits
- Remote-first company, with this role intended for in-person collaboration at the San Francisco, CA office
- Competitive compensation with meaningful ownership
- Flexible PTO
- 401k
- Wellness benefits, including free therapy sessions
- Technology and work-from-home reimbursement
- Flexible work schedules
Tech Stack
Categories
About Fieldguide
Fieldguide is a leading AI-native platform transforming how audit, advisory, and assurance work gets done. Purpose-built for regulated environments, Fieldguide brings professionals and agentic AI together to modernize engagement delivery from planning through close, enabling audit and advisory practitioners to deliver higher-quality work, strengthen client experiences, and unlock growth for the decade ahead. Fieldguide is trusted by leading firms including KPMG, RSM US LLP, Baker Tilly, CBIZ, BDO, Grant Thornton, Forvis Mazars, CLA (CliftonLarsonAllen), CohnReznick, Wipfli LLP, Carr Riggs & Ingram, Cherry Bekaert, Aprio LLP, UHY Advisors Inc., Weaver, LBMC, Elliott Davis, Frazier & Deeter, Cohen & Co, BerryDunn, Schellman, and Warren Averett, among others. Fieldguide is also the recipient of the Accounting Today Top New Products Award as well as a five time winner of the CPA Practice Advisor Technology Innovation Award. Fieldguide is headquartered in San Francisco and backed by Growth Equity at Goldman Sachs Alternatives, Bessemer Venture Partners, 8VC, Thomson Reuters, and other leading investors. Fieldguide announced its $75m series C in February 2026 and has raised $125m to date.