Machine Learning Scientist
Arena Intelligence, Inc.5 months ago
Responsibilities
- Design and conduct experiments to evaluate AI model behavior across various dimensions.
- Develop new metrics, methodologies, and evaluation protocols beyond traditional benchmarks.
- Analyze large-scale human voting and interaction data for insights into model performance.
- Collaborate with engineers to implement and scale research findings into production systems.
- Prototype and test research ideas rapidly while balancing rigor with iteration speed.
- Author internal reports and external publications for the broader ML research community.
- Partner with model providers to shape evaluation questions and support responsible testing.
- Contribute to the scientific integrity and transparency of the Arena Intelligence leaderboard.
Requirements
- PhD or equivalent research experience in Machine Learning, NLP, Statistics, or a related field.
- Strong understanding of LLMs and modern deep learning architectures.
- Proficiency in Python and ML research libraries such as PyTorch, JAX, or TensorFlow.
- Demonstrated ability to design and analyze experiments with statistical rigor.
- Experience publishing research or working on open-source projects in ML, NLP, or AI evaluation.
- Comfortable working with real-world usage data and designing metrics beyond standard benchmarks.
- Ability to translate research questions into practical systems and collaborate across teams.
- Passion for open science, reproducibility, and community-driven research.
Benefits
- Competitive compensation and equity aligned to market locations.
- Comprehensive health and wellness benefits, including medical, dental, and vision.
- Opportunity to work on cutting-edge AI with a small, mission-driven team.
- Culture that values transparency, trust, and community impact.