Staff Machine Learning Research Scientist, LLM Evals

Scale AI

4 months ago

New York, NY, USA +2 more

Staff+

H1B Sponsor

Base Salary

$240k - $380k/yr

Responsibilities

Drive research on the effectiveness and limitations of existing LLM evaluation techniques.
Design and develop novel evaluation benchmarks for large language models.
Communicate and collaborate with clients and peer teams for cross-functional projects.
Refine metrics and create standardized evaluation protocols with internal teams and external partners.
Implement scalable and reproducible evaluation pipelines using modern ML frameworks.
Publish research findings in top-tier AI conferences and contribute to open-source initiatives.
Mentor and guide research scientists and engineers, providing technical leadership.
Stay engaged with the ML research community and track emerging work.

5+ years of hands-on experience in large language model, NLP, and Transformer modeling.
Proven track record of major research impacts in a fast-paced environment.
Experience leading a team of research scientists and engineers.
Excellent written and verbal communication skills.
Published research in major machine learning conferences or journals.
Previous experience in a customer-facing role.