8 hours ago
Base Salary
$240k - $290k/yr
Responsibilities
- Own the evaluation platform end to end, including tooling and systems for generating, annotating, reviewing, and adapting model evaluations at scale.
- Design CLIs, APIs, GUIs, and storage layers for a seamless and collaborative evaluation workflow.
- Partner with research teams in video, image, audio, agents, and robotics to build and generalize evaluation capabilities.
- Set standards for reproducibility, metric definitions, reporting formats, and trustworthy evaluation results.
- Drive adoption and integration of the platform across training, production model serving, and broader research efforts.
- Contribute to ML Platform tools and systems used to train and serve frontier models.
Requirements
- 5+ years of experience building ML infrastructure or data platforms in production, including experience with evaluation, experimentation, or benchmarking systems.
- Strong Python and PyTorch skills, with hands-on experience running large-batch GPU workloads on Kubernetes.
- Experience designing data pipelines and storage for large volumes of media or model outputs, including versioning and reproducibility.
- Familiarity with experimental statistics, including paired comparisons, confidence intervals, multiple-comparison pitfalls, and inter-rater agreement.
- Experience building internal tools end to end from the command line to the browser.
- Ability to lead a broad technical area, gather requirements, set direction, make tradeoffs, and drive a roadmap.
- Familiarity with the full model development lifecycle across data, training, evaluation, and serving.
- Preferred experience building evaluation suites for generative models, LLM- or VLM-as-judge pipelines, online experimentation platforms, or evaluations for agents and robotics policies.
- Ability to work independently with research teams, move quickly, and apply strong systems thinking to production reliability.
Benefits
- The posting describes a comprehensive compensation package and an emphasis on pay equity, but does not provide a specific salary amount.
- Employees have access to models, agents, GPUs, storage, and cloud services for building evaluation pipelines.
- Runway is committed to an inclusive workplace and equal opportunity for employees from diverse backgrounds.
Tech Stack
Categories
About Runway
Runway builds generative video and multimodal AI tools—text-to-video, image-to-video, advanced video editing, and real-time agents—delivered via web apps and APIs for creators, studios, and enterprises. Its business model centers on subscriptions, usage-based credits, and licensing; notable releases include Gen-4.5, the GWM-1 world model, and a robotics SDK. Founded in 2018 and headquartered in New York, the company is privately held.