2 days ago
Base Salary
$180k - $450k/yr
Responsibilities
- Build and maintain evaluation harnesses for held-out loss, data-domain performance, and text, code, math, and reasoning capabilities.
- Design checkpoint and smaller-model evaluations using log-likelihood scoring, few-shot prompts, and generated completions.
- Assess benchmark coverage and saturation, quantify uncertainty, and detect training-data contamination.
- Version datasets, prompts, graders, and inference settings for reproducible comparisons across checkpoints and training runs.
- Analyze experiments with researchers to improve training recipes, select checkpoints, and prioritize capability improvements.
- Develop new benchmarks and extend evaluation tools to image, video, speech, and audio tasks.
Requirements
- Experience building model evaluation systems or experimental tools for reproducible comparisons across models or checkpoints.
- Knowledge of language-model evaluation methods, including log-likelihood and generative scoring, prompt sensitivity, few-shot variance, and formatting effects.
- Strong statistical judgment covering uncertainty estimation, sample-size planning, and statistical versus practical significance.
- Experience identifying benchmark contamination, saturation, or gaps between benchmark scores and intended capabilities.
- Strong Python engineering skills for reliable evaluation pipelines running efficiently across many checkpoints.
- A record of developing benchmarks, evaluation methods, or tools that inform research and engineering decisions.
- Preferred qualifications include experience with human or model-based graders, capability-scaling studies, code execution sandboxes, math verifiers, pretraining or mid-training evaluation, multimodal evaluation, and constrained inference budgets.
Tech Stack
Categories
AI Research
