4 hours ago
Singapore (city not specified)Mid Level
Base Salary
$100k - $170k/yr
Responsibilities
- Design, implement, and maintain internal benchmarks for evaluating frontier AI agents on domain-specific tasks.
- Partner with subject-matter experts to translate real-world workflows into benchmark tasks and evaluation criteria.
- Build reliable infrastructure for running models and agents against benchmark tasks at scale.
- Develop metrics and analyses to assess benchmark difficulty, reliability, and failure modes.
- Validate benchmark results against real-world evaluations and user needs.
- Write technical documentation and reports explaining benchmark methods and results.
Requirements
- Two to four years of experience in software engineering, machine learning engineering, or research engineering, including at least two years building AI benchmarks, evaluation infrastructure, or agent environments.
- Technical education or strong fundamentals in a technical field.
- Proficiency with Python, Docker, and Linux for research or production infrastructure.
- Hands-on experience designing and operating evaluations or environments for AI agents or large language models.
- Experience building reliable model or agent execution infrastructure, developing evaluation metrics or statistical analyses, and validating benchmark performance.
- Ability to work with subject-matter experts and understand workflows across technical or business domains.
- Strong technical writing, attention to detail, independent problem-solving, and comfort in an early-stage environment.
- Published research or technical writing on AI benchmarks, evaluation methods, or model failure modes is valued; reinforcement learning pipelines, data generation, and agent evaluation experience are also relevant.
Benefits
- Base salary of $100,000 to $170,000 USD annually.
- Visa sponsorship is available.
- On-site work in Singapore, Singapore.