Member of Technical Staff - Pre-Training Evals Infra
Reflection22 hours ago
London, United Kingdom +2 moreSenior
Responsibilities
- Design and evolve APIs, abstractions, and data models for evaluation tasks, datasets, configurations, model outputs, and scores.
- Build scalable distributed infrastructure to evaluate model checkpoints during training and before release, including orchestration, observability, and failure recovery.
- Optimize inference and distributed execution for throughput, GPU utilization, and time to results.
- Build validation and regression tests and preserve dataset, configuration, and result provenance for reproducibility.
- Create researcher-facing interfaces and dashboards for inspecting outputs, comparing checkpoints and experiments, and debugging unexpected behavior.
- Operate and monitor evaluation workloads, triage failures, and develop alerts and diagnostics.
- Partner with researchers to identify bottlenecks and convert bespoke workflows into reusable self-service tools.
Requirements
- Strong proficiency in Python, C++, or Rust, with deep understanding of concurrency, memory behavior, and performance.
- Experience designing, building, and operating reliable, maintainable distributed systems.
- Working knowledge of model inference and evaluation and how implementation and configuration choices affect measured model quality.
- Ability to investigate complex problems across system boundaries and identify root causes.
- Strong product judgment and experience translating technical-user needs into clear APIs and usable tools.
- Ability to own ambiguous problems end to end while balancing delivery speed, reliability, and maintainability.
- Persistence and attention to detail when investigating discrepancies in model outputs, evaluation results, and system behavior.
Benefits
- Top-tier compensation and equity structured for global talent; specific salary figures are not provided.
- Stock options for all employees.
- Comprehensive medical, dental, vision, and life insurance with an annual wellness allowance.
- Lunch and dinner provided daily in the office.
- 22 weeks of paid parental leave for birthing, non-birthing, adoptive, and surrogate parents.
- Unlimited paid time off in the U.S. and 30 vacation days in the U.K.
- Visa sponsorship and support for long-term immigration pathways where applicable.
- Regular off-sites, happy hours, and team celebrations.
- The role may be subject to U.S. export-control access requirements and government authorization.
Categories
About Reflection
Reflection is a New York–based, privately held research lab developing open foundational AI models and agentic coding tools for developers, enterprises, and public-sector users. The team includes former researchers from DeepMind, OpenAI, and Anthropic, and their work focuses on transparent, customizable systems that organizations can deploy with ownership and control.