12 hours ago
Singapore, SingaporeStaff+
Base Salary
$150k - $250k/yr
Responsibilities
- Lead the data quality team in building evaluation systems for tasks across reinforcement-learning environments, synthetic data, benchmarks, and domain-specific workflows.
- Define data quality strategy, quality-control systems, standards, metrics, experiments, and agent-output grading methods.
- Develop scalable validation methods for synthetic data, including failure-mode analysis, task mutation checks, and trajectory auditing.
- Partner with research engineers, domain experts, and data vendors to diagnose quality issues and improve data-generation workflows.
- Translate qualitative research insights and domain judgment into internal tools, dashboards, validation pipelines, and feedback loops.
- Develop evaluation intuition around realistic, learnable, diverse, reliable, and useful agent-training data.
- Mentor research engineers and maintain high standards for technical rigor, clarity, and execution.
Requirements
- 5+ years of experience in research or data quality engineering focused on AI/ML data evaluation systems.
- Track record leading technical projects or teams through definition, implementation, and iteration on ambiguous problems.
- Advanced proficiency in Python, Docker, and Linux environments.
- Background as an ML researcher or engineer specializing in the quality and evaluation of AI training data at scale.
- Experience designing metrics, experiments, and QA/QC processes.
- Experience translating research insights into production systems and data pipelines.
- Experience working with subject-matter experts to convert domain judgment into scalable review or generation systems.
- Strong written communication skills for explaining methodology to researchers, engineers, and external audiences.
- Research-oriented understanding of AI evaluations and post-training beyond surface-level agent tooling.
- Comfort navigating systems involving domain experts, vendors, model outputs, graders, and infrastructure.
- Early-stage startup experience and ability to work independently at pace.
Benefits
- Annual salary of USD 150,000 to 250,000.
- Visa sponsorship is available.
- On-site role based in the Singapore office; candidates must be available to work from Singapore.