1 day ago
Singapore, SingaporeMid Level
Base Salary
$100k - $170k/yr
Responsibilities
- Design, implement, and maintain benchmarks for evaluating AI agents on domain-specific tasks.
- Translate real workflows into practical benchmark tasks and evaluation criteria with subject-matter experts.
- Build and operate infrastructure to run models and agents against benchmark tasks at scale.
- Develop metrics and statistical analyses for benchmark difficulty, reliability, and failure modes.
- Validate benchmark results against real-world evaluations and user needs.
- Write technical documentation and reports explaining methods and results.
Requirements
- Two to four years of experience in software engineering, machine learning engineering, or research, including at least two years building AI benchmarks, evaluation infrastructure, or agent environments.
- Hands-on experience designing and running evaluations or benchmark environments for AI agents or large language models.
- Strong proficiency with Python, Docker, and Linux, including building and operating reliable infrastructure.
- Experience developing metrics, statistical analyses, or validation studies for evaluation systems.
- Experience collaborating with subject-matter experts and understanding workflows across technical or business domains.
- Strong technical writing, attention to detail, and ability to work independently in an early-stage environment.
- Research publications or technical writing on benchmarking, model evaluation, or failure modes are beneficial.
- Experience with reinforcement learning pipelines or public benchmark projects is useful.
Benefits
- Base salary of $100,000 to $170,000 USD annually.
- Visa sponsorship is available.
- On-site work in Singapore, Singapore.