13 hours ago
Base Salary
$150k - $180k/yr
Responsibilities
- Lead the data-quality team in building systems that evaluate thousands of tasks across RL environments, synthetic-data pipelines, benchmarks, and domain-specific workflows.
- Define the data-quality strategy by building QC systems, enforcing standards, and designing experiments to grade agent outputs.
- Develop scalable methods for validating synthetic data, including failure-mode analysis, task-mutation checks, and trajectory auditing.
- Partner with research engineers, domain experts, and data vendors to diagnose quality issues and improve data-generation workflows.
- Translate qualitative research insights into internal tools, dashboards, validation pipelines, and feedback loops.
- Establish standards for realistic, learnable, diverse, reliable, and useful agent-training data.
- Mentor research engineers and maintain high standards for technical rigor, clarity, and execution speed.
Requirements
- 5 or more years of experience in research or data-quality engineering focused on AI/ML data evaluation.
- Demonstrated experience leading technical teams or projects in data quality or AI/ML evaluation from problem definition through implementation and iteration.
- Advanced proficiency in Python, Docker, and Linux environments.
- Experience building QC systems, evaluations, benchmarks, synthetic-data pipelines, or model-evaluation infrastructure.
- Deep research-oriented understanding of AI evaluations and post-training beyond surface-level agent-harness projects.
- Ability to design metrics, experiments, and QA/QC processes and recognize characteristics of high-quality training data.
- Experience collaborating with subject-matter experts to convert domain judgment into scalable review or generation systems.
- Strong written communication skills for explaining methodology to technical and non-technical audiences.
- Ability to work independently in an early-stage startup with ambiguous, fast-moving priorities.
- Detail-oriented approach with attention to subtle inconsistencies and data edge cases.
Benefits
- Salary of $150,000 to $180,000 per year (USD).
- Visa sponsorship is available.
- On-site role in the United States; candidates in or willing to relocate to the San Francisco Bay Area are preferred.