14 days ago
Base Salary
$500k - $850k/yr
Responsibilities
- Design and run evaluations of Claude’s capabilities, behavior, knowledge, and safety properties.
- Build and harden distributed evaluation execution infrastructure for reliable large-scale checkpoint testing.
- Own dashboards for monitoring model health during training and improve their signal-to-noise ratio, latency, and regression detection.
- Debug anomalous evaluation results during live training runs and determine whether issues originate in the model, harness, data, or infrastructure.
- Improve evaluation tooling, libraries, and researcher workflows.
- Partner with research teams to define measurable capability criteria and interpret results throughout training.
- Run experiments on how prompting, sampling, and scaffolding choices affect benchmark results.
- Communicate evaluation methods and findings to internal stakeholders and appropriate external audiences.
Requirements
- Strong Python programming skills, including production or research infrastructure experience.
- Experience building or operating reliable distributed systems, data pipelines, or other infrastructure at scale.
- Clear written and verbal communication skills, including explaining technical results to non-specialists.
- Comfort operating in an on-call or production-support capacity during live training runs.
- Interest in the societal impacts of AI and in steering powerful AI systems toward safe and beneficial outcomes.
- Bachelor’s degree in a relevant field, or an equivalent combination of education, training, and/or experience.
- Preferred qualifications include hands-on Claude or large-language-model experience, data visualization and trusted dashboard development, language-model evaluation metrics, observability or experiment tracking, statistics, experimental design, large-scale dataset sourcing and processing, and ML training infrastructure experience.
Benefits
- Annual compensation range of $500,000–$850,000 USD.
- Hybrid policy requiring staff to work from an office at least 25% of the time, with some roles requiring more.
- Visa sponsorship may be available, with immigration-lawyer support.
- Competitive benefits, optional equity donation matching, generous vacation and parental leave, flexible working hours, and office collaboration space.
Tech Stack
Categories
About Anthropic
We're an AI research company that builds reliable, interpretable, and steerable AI systems. Our first product is Claude, an AI assistant for tasks at any scale. Our research interests span multiple areas including natural language, human feedback, scaling laws, reinforcement learning, code generation, and interpretability.