1 day ago
Responsibilities
- Own the internal evaluation suite and infrastructure end to end, keeping it reliable, optimized, reproducible, and scalable.
- Design held-out evaluations for capabilities including agentic workflows, computer use, and cybersecurity, with documented construction and contamination checks.
- Build evaluations targeted at small edge models and debug unexpected scores by reviewing individual generated outputs.
- Convert customer and open-source feedback into environments for data, training, and evaluation, and partner with other teams on their evaluation needs.
- Release open-source evaluations, write papers, and publish failure-mode reviews of popular evaluations.
- Drive post-training decisions through internal evaluations and help co-own the evaluation function.
Requirements
- Strong evaluation-design ability, including translating high-level capabilities into accurate, general-purpose, hard-to-game evaluations that distinguish model quality.
- Prior experience building evaluations that demonstrate coverage, reproducibility, contamination handling, and useful signal, including at least one benchmark, evaluation paper, or substantial contribution built from scratch.
- Experience with evaluation fundamentals including contamination, saturation, judge bias, and variance.
- Hands-on experience running and managing evaluations within an evaluation, training, engineering, or DevOps team.
- Strong software engineering skills sufficient to own evaluation infrastructure and debug inference and infrastructure failures at scale independently.
- Helpful qualifications include model training experience, publications or research experience, deep systems expertise, and familiarity with small edge models.
Benefits
- Full-time hybrid role based in San Francisco.
- Competitive base salary plus equity; final offers reflect skills, experience, and location.
- Unlimited paid time off and company-wide Refill Days and long weekends.
- Comprehensive health coverage, with details varying by location.
- Paid leave for pregnancy, parenting, medical needs, and caring for family.
- Real compute for running evaluations at scale and the opportunity to open-source and publish evaluation work.
Tech Stack
Helm
Categories
About Liquid AI
Liquid AI builds general-purpose AI systems that run efficiently from data center accelerators to on-device hardware, emphasizing low latency, memory efficiency, privacy, and reliability. The company partners with enterprises in consumer electronics, automotive, life sciences, and financial services to deploy and benchmark models for real-world workloads. Founded in 2023 out of MIT CSAIL and headquartered in Cambridge, Massachusetts, it is privately held.