12 hours ago
Singapore, SingaporeMid Level / Senior
H1B sponsor
Base Salary
$150k - $250k/yr
Responsibilities
- Automate quality control for training data created by companies using the platform infrastructure.
- Define and enforce quality standards for reinforcement-learning training data.
- Build scalable QC, data-validation, auditing, sampling, and rule-based or model-assisted validation pipelines.
- Design experiments, benchmarks, rubrics, and metrics to grade agent outputs and evaluate training-data quality.
- Partner with external data vendors to debug quality issues, diagnose agent failure modes, and improve data-generation processes.
- Integrate QC learnings into infrastructure tools and the vendor portal to reduce anomalies, inconsistencies, and edge cases.
Requirements
- 2 to 4 years of experience in research engineering or a similar QC automation role.
- Proficiency in Python, Docker, and Linux environments.
- Experience building scalable data-validation pipelines and automated QA/QC systems end-to-end.
- Experience with benchmarks and evaluations for reinforcement-learning training data, including realistic tasks, reliable rubrics, and useful trajectories.
- Experience creating QC systems based on human judgment rather than heavy reliance on large language models.
- Experience designing experiments and metrics to grade agent outputs.
- Experience partnering with data vendors to debug quality issues and provide actionable feedback.
- Strong knowledge of statistics and comfort designing metrics and QA/QC processes.
- Strong written and verbal communication skills and the ability to work autonomously in an unstructured, fast-paced environment.
Benefits
- On-site role in Singapore.
- Candidates based in San Francisco or working remotely as independent contractors, particularly from Europe, may also be considered.