3 hours ago
Base Salary
$300k - $350k/yr
Responsibilities
- Own evaluation frameworks and infrastructure measuring the accuracy, quality, and reliability of AI agents across workflows, assistants, document processing, and computer use.
- Develop golden datasets, benchmarks, and regression suites with Forward Deployed Engineers and domain experts for real-world insurance tasks and edge cases.
- Design evaluations for multi-step agent trajectories, tool use, reasoning, and end-to-end task completion.
- Build LLM-as-judge evaluations, deterministic graders, human evaluation workflows, and quality rubrics.
- Integrate offline and online evaluations into development and release processes to measure improvements and catch regressions.
- Benchmark frontier and open-weight models across accuracy, latency, cost, and reliability to inform model selection and production releases.
- Create continuous improvement loops using production failures, customer feedback, and agent traces.
- Build reusable evaluation frameworks, benchmarking systems, and production pipelines for insurance workflows and AI skills.
Requirements
- 5+ years of software engineering experience with strong hands-on experience building production AI/ML systems.
- Experience building evaluation frameworks, LLM-based applications, model benchmarking systems, or applied AI infrastructure.
- Strong understanding of modern LLMs, agent architectures, evaluation methodologies, and model performance tradeoffs.
- Strong Python engineering skills and ability to build reliable, production-grade systems.
- Ability to independently own ambiguous technical problems from research and experimentation through production.
- Clear written and verbal communication skills.
- Preferred experience with LLM-as-judge systems, human evaluation, agent trajectory evaluations, or large-scale benchmarking.
- Preferred experience with open-weight models, fine-tuning, post-training, or model routing.
- Preferred B2B SaaS background and 0-to-1 experience.
Benefits
- Fully covered health, dental, and vision benefits.
- Competitive compensation, meaningful stock options, and company 401(k).
- Unlimited PTO.
- Fully in-office in San Francisco five days per week.
- Lunch and dinner provided onsite.
- Team events including happy hours and off-sites.
- Pre-tax commuter benefits.
Tech Stack
Categories
About FurtherAI
FurtherAI builds domain-specific AI software for insurers, reinsurers, MGAs, and brokers, offering an AI Workspace that automates submission intake, policy comparisons, and underwriting audits. It sells an enterprise platform that augments internal teams with AI coworkers to streamline underwriting and operations. Founded in 2023 and headquartered in San Francisco, the privately held company is venture-backed (including Y Combinator) and supports customers writing over $15B in premiums across all 50 states.