1 day ago
Base Salary
$180k - $500k/yr
Responsibilities
- Design, implement, and maintain benchmarks and metrics for tool use, agentic behavior, and real-world reasoning.
- Build and operate end-to-end LLM evaluation systems, including runs, scoring, dashboards, and reporting.
- Conduct systematic failure analysis of model outputs and translate findings into improvements for reward design, data curation, and benchmark design.
- Create and refine rubrics, automated evaluators, and scoring frameworks, including human-versus-model-as-judge calibration and agreement analysis.
- Quantify data usability, quality, and impact on benchmarks to guide data generation, augmentation, and curation.
- Collaborate with AI researchers, applied AI teams, and data producers to align evaluations with training objectives.
- Own benchmarking, evaluation, and failure-analysis workflows in a fast-paced research environment.
Requirements
- Strong applied research background focused on model evaluation, benchmarking, and/or failure analysis.
- Strong coding skills and hands-on experience with ML models and evaluation code.
- Solid understanding of data structures, algorithms, and backend systems.
- Familiarity with APIs, SQL/NoSQL, and cloud platforms for running and storing evaluation results.
- Ability to reason about model behavior, experimental results, and data quality from evaluations and failure analyses.
- Willingness to work in person in San Francisco five days per week in a high-intensity, high-ownership environment.
- Preferred: industry experience on a post-training or evaluation/benchmarking team.
- Preferred: publications at top-tier venues such as NeurIPS, ICML, or ACL.
- Preferred: experience with LLM evaluations, benchmarks, failure-analysis pipelines, synthetic data generation, rubric design, or RL-style reward-shaping workflows.
- Preferred: relevant work samples or code, such as evaluation frameworks, benchmark suites, failure-analysis reports, or tooling.
Benefits
- $1.5K monthly meal stipend.
- Free Equinox membership.
- $200 monthly laundry reimbursement.
- $200 monthly personal wellness reimbursement.
- In-person work five days per week in the San Francisco, New York City, or London office.
Tech Stack
Categories
About Mercor
Mercor builds an expert-powered platform that trains, evaluates, and deploys AI systems for AI labs and enterprises. It operates APEX to assess model performance on economically valuable tasks, and provides services such as custom AI agents and teams of vetted domain experts who encode organizational knowledge into AI. Founded in 2023 and headquartered in San Francisco, it is privately held and works with frontier AI labs and enterprise clients requiring strict data isolation.