2 months ago
Base Salary
$500k - $850k/yr
Responsibilities
- Design and run evaluations of Claude’s capabilities, behavior, knowledge, and safety properties.
- Build and harden distributed evaluation execution infrastructure for reliable large-scale checkpoint testing.
- Own dashboards for monitoring model health during training and improve their signal-to-noise ratio, latency, and regression detection.
- Debug anomalous evaluation results during live training runs and determine whether issues originate in the model, harness, data, or infrastructure.
- Improve evaluation tooling, libraries, and researcher workflows.
- Partner with research teams to define measurable capability criteria and interpret results throughout training.
- Run experiments on how prompting, sampling, and scaffolding choices affect benchmark results.
- Communicate evaluation methods and findings to internal stakeholders and appropriate external audiences.
Requirements
- Strong Python programming skills, including production or research infrastructure experience.
- Experience building or operating reliable distributed systems, data pipelines, or other infrastructure at scale.
- Clear written and verbal communication skills, including explaining technical results to non-specialists.
- Comfort operating in an on-call or production-support capacity during live training runs.
- Interest in the societal impacts of AI and in steering powerful AI systems toward safe and beneficial outcomes.
- Bachelor’s degree in a relevant field, or an equivalent combination of education, training, and/or experience.
- Preferred qualifications include hands-on Claude or large-language-model experience, data visualization and trusted dashboard development, language-model evaluation metrics, observability or experiment tracking, statistics, experimental design, large-scale dataset sourcing and processing, and ML training infrastructure experience.
Benefits
- Annual compensation range of $500,000–$850,000 USD.
- Hybrid policy requiring staff to work from an office at least 25% of the time, with some roles requiring more.
- Visa sponsorship may be available, with immigration-lawyer support.
- Competitive benefits, optional equity donation matching, generous vacation and parental leave, flexible working hours, and office collaboration space.
Tech Stack
Categories
About Anthropic
Anthropic builds large language models and the Claude AI assistant for developers and enterprises, offered via API access and enterprise plans. Founded in 2021 and headquartered in San Francisco, it distributes Claude through its own platform and via partners such as Amazon Bedrock and Google Cloud’s Vertex AI. Its work emphasizes model reliability, interpretability, and practical tooling for tasks like coding assistance, analysis, and customer support automation.