Research Engineer, Benchmarks

11 hours ago

Singapore, SingaporeMid Level / Senior

H1B sponsor

Base Salary

$150k - $250k/yr

Responsibilities

Design, implement, and maintain internal benchmarks for evaluating frontier AI agents on domain-specific tasks.
Partner with subject-matter experts to define realistic workflows and evaluation criteria.
Build and operate infrastructure for running models and agents against benchmark tasks at scale.
Develop metrics and statistical analyses for benchmark difficulty, reliability, and failure modes.
Validate benchmark performance against real-world evaluations and customer needs.
Write technical documentation and benchmark reports for research and engineering audiences.

Require 2 to 4 years of experience in research engineering or ML engineering focused on AI benchmarks, evaluation infrastructure, or agent environments.
Require strong proficiency in Python, Docker, and Linux.
Require experience designing and running benchmarks or evaluation environments for AI agents or large language models.
Require experience building reliable infrastructure for running AI models or agents against evaluation tasks at scale.
Require experience developing metrics or validation studies for benchmark difficulty, reliability, and real-world correlation.
Require the ability to collaborate with domain experts and translate complex workflows into evaluation criteria.
Require strong attention to detail and the ability to identify inconsistencies and edge cases.
Require comfort working independently in fast-paced, early-stage startup environments with unstructured problem spaces.
Require excellent written communication skills for technical documentation and cross-timezone collaboration.
Published papers or technical writing on AI benchmarking, model evaluation, or failure modes is a strong plus.
Experience with reinforcement learning training pipelines, data generation, or RL agent evaluation is a plus.