2 days ago
Singapore, SingaporeMid Level
Base Salary
$100k - $170k/yr
Responsibilities
- Design, implement, and maintain benchmarks for evaluating frontier AI agents on domain-specific tasks.
- Collaborate with subject-matter experts to convert real workflows into benchmark tasks and evaluation criteria.
- Build reliable infrastructure to run AI models and agents against evaluation tasks at scale.
- Develop metrics and analyses for benchmark difficulty, reliability, and failure modes.
- Validate benchmark results against real-world evaluations and technical user needs.
- Write technical documentation and reports for research and engineering audiences.
Requirements
- At least two years of experience in research engineering or machine learning engineering focused on AI benchmarks, evaluation infrastructure, or agent environments.
- Experience designing, implementing, and operating benchmarks or evaluation environments for AI agents or large language models.
- Proficiency in Python, Docker, and Linux environments.
- Experience building reliable infrastructure for running AI models or agents and collaborating with subject-matter experts.
- Experience analyzing technical or business workflows and developing metrics or validation studies for benchmark quality.
- Strong technical writing, attention to detail, and ability to work independently in unstructured environments.
- Experience with reinforcement learning pipelines, data generation, published work, or public benchmarks is beneficial.
Benefits
- Annual salary range of $100,000 to $170,000 USD.
- Visa sponsorship is available.
- On-site work in Singapore, Singapore.