2 days ago
Base Salary
$130k - $225k/yr
Responsibilities
- Lead a team developing systems to evaluate large volumes of tasks across environments, datasets, benchmarks, and specialized workflows.
- Define data quality strategy, quality-control systems, standards, and experiments for assessing agent outputs.
- Develop scalable synthetic-data validation approaches, including failure-mode analysis, task mutation checks, and trajectory auditing.
- Collaborate with research engineers, domain experts, and data vendors to diagnose quality issues and improve data generation.
- Translate research findings into production tools, dashboards, validation pipelines, and feedback loops.
- Establish standards for useful agent training data and mentor research engineers.
Requirements
- At least 5 years of research or engineering experience building AI or machine learning data evaluation and quality systems.
- Experience leading technical teams or projects from problem definition through implementation and iteration.
- Advanced proficiency in Python, Docker, and Linux environments.
- Strong background in AI evaluations, post-training, synthetic data validation, benchmarks, or related evaluation infrastructure.
- Experience designing metrics, experiments, and quality processes and translating research insights into production systems and pipelines.
- Ability to work with subject-matter experts to create scalable review or data-generation systems.
- Clear written communication, careful attention to detail, sound judgment about training-data quality, and experience in an early-stage startup environment.
Benefits
- Visa sponsorship is available.
- The role is on-site in the United States.