1 day ago
Singapore, SingaporeMid Level / Senior
Base Salary
$150k - $250k/yr
Responsibilities
- Automate quality control for training data created by companies using the platform infrastructure.
- Build QC systems grounded in human judgment and define and enforce training-data quality standards.
- Design experiments, metrics, benchmarks, rubrics, and evaluation processes to grade agent outputs and trajectories.
- Partner with data vendors to debug quality issues, diagnose agent failure modes, and improve data-generation processes.
- Build auditing systems using sampling strategies and rule-based or model-assisted validation pipelines.
- Integrate QC insights into infrastructure tools and vendor-facing systems to reduce anomalies, inconsistencies, and edge cases.
Requirements
- 2 to 4 years of experience in engineering or research roles, ideally focused on QC automation or data quality.
- Proficiency in Python, Docker, and Linux environments.
- Experience building scalable data-validation pipelines and automated QA/QC systems end-to-end without a fully prescribed roadmap.
- Experience with benchmarks and evaluations for reinforcement-learning training data, including realistic tasks, reliable rubrics, and useful trajectories.
- Experience creating QC systems based on human judgment rather than LLM-centric approaches.
- Experience designing experiments and metrics to grade agent outputs and providing actionable feedback to data vendors.
- Solid knowledge of statistics and comfort designing metrics and QA/QC processes.
- Strong written and verbal communication skills, curiosity, independence, and comfort working in unstructured, fast-paced startup environments.
Benefits
- Salary range of $150,000 to $250,000 USD annually.
- Visa sponsorship is available.
- Primary work arrangement is on-site in Singapore; candidates in San Francisco or fully remote independent contractors, particularly from Europe, may also be considered depending on location.