Senior AI Engineer, Quality
Fieldguide2 months ago
Remote, United States or San Francisco, CA, USASenior / Mid Level
H1B Sponsor
Base Salary
$200k - $250k/yr
Responsibilities
- Design and build a unified evaluation platform for agentic systems and audit workflows.
- Create observability systems to monitor agent behavior and production failures.
- Develop automated pipelines for rapid model evaluation.
- Implement comparison frameworks to measure model effectiveness and quality.
- Define evaluation standards and advocate for evaluation-driven development.
- Collaborate with product and ML engineers to integrate evaluation requirements.
Requirements
- Multiple years of experience shipping production software in complex systems.
- Proficiency in TypeScript, React, Python, and Postgres.
- Experience deploying LLM-powered features in production.
- Familiarity with evaluation frameworks for model outputs.
- Knowledge of observability or tracing infrastructure for AI/ML systems.
- Comfort operating in ambiguity and taking responsibility for outcomes.
Benefits
- Competitive compensation packages with meaningful ownership.
- Flexible PTO.
- 401k.
- Wellness benefits, including free therapy sessions.
- Technology and work from home reimbursement.
- Flexible work schedules.