3 days ago
Base Salary
$240k - $290k/yr
Responsibilities
- Own the evaluation platform end to end, including systems for generating, annotating, reviewing, and adapting model outputs at frontier scale.
- Design CLIs, APIs, GUIs, and storage layers for fast, collaborative evaluation workflows.
- Work with research teams across video, image, audio, agents, and robotics to understand evaluation needs and generalize solutions into the platform.
- Set standards for reproducibility, metric definitions, reporting formats, and trustworthy model-evaluation results.
- Drive adoption and integration of the platform across training, production model serving, and other research efforts.
- Contribute to ML Platform tools and systems used to train and serve frontier models.
Requirements
- 5+ years of experience building ML infrastructure or data platforms in production, including experience with evaluation, experimentation, or benchmarking systems.
- Strong Python and PyTorch skills, with hands-on experience running large batch GPU workloads on Kubernetes.
- Experience designing data pipelines and storage for large volumes of media or model outputs, including versioning and reproducibility.
- Familiarity with experimental statistics, including paired comparisons, confidence intervals, multiple-comparison pitfalls, and inter-rater agreement.
- Experience building internal tools end to end from the command line to the browser.
- Ability to lead a broad technical area, gather requirements, set direction, make tradeoffs, and drive a roadmap.
- Familiarity with the full model development lifecycle spanning data, training, evaluation, and serving.
- Ability to work independently with research teams, move quickly, and apply systems thinking to production reliability.
- Preferred: experience building evaluation suites for generative models, LLM- or VLM-as-judge pipelines, online experimentation platforms, offline-to-product metric connections, or evaluation systems for agents or robotics policies.
Tech Stack
Categories
About Runway
Runway builds generative video and multimodal AI tools—text-to-video, image-to-video, advanced video editing, and real-time agents—delivered via web apps and APIs for creators, studios, and enterprises. Its business model centers on subscriptions, usage-based credits, and licensing; notable releases include Gen-4.5, the GWM-1 world model, and a robotics SDK. Founded in 2018 and headquartered in New York, the company is privately held.