Research Engineer, Model Evaluations

Anthropic

3 months ago

New York, NY, USA or San Francisco, CA, USA

Mid Level / Senior / Staff+

H1B Sponsor

Base Salary

$300k - $405k/yr

Responsibilities

Design novel evaluation methodologies to assess model capabilities across diverse domains.
Lead the design and architecture of Anthropic's evaluation platform.
Implement and maintain high-throughput evaluation pipelines during production training.
Analyze evaluation results to identify patterns and opportunities for model improvement.
Partner with research teams to develop domain-specific evaluations.
Build infrastructure for rapid iteration on evaluation design.
Establish best practices and standards for evaluation development.
Mentor team members and contribute to evaluation expertise growth.
Coordinate evaluation efforts during critical training runs.
Contribute to research publications and external communications.

Experience designing and implementing evaluation systems for machine learning models.
Demonstrated technical leadership experience in complex technical projects.
Skilled in systems engineering and experimental design.
Strong programming skills in Python and experience with distributed computing frameworks.
Ability to translate between research needs and engineering constraints.
Results-oriented and thrive in fast-paced environments.
Enjoy collaborative work and can communicate technical concepts effectively.
Care about AI safety and societal impacts.
Experience with statistical analysis and large-scale experimental data.

Python