12 hours ago
San Francisco, CA, USAMid Level
Responsibilities
- Convert licensed production codebases into reproducible, containerized coding environments.
- Create coding tasks from issues, pull requests, commit histories, and incident records, including bug fixes, features, refactors, testing, and multi-file agent work.
- Build and improve verifiers using hidden tests, rubric-based or model-based grading, and safeguards against reward hacking and answer leakage.
- Run model rollouts, review traces, and repair or remove broken, ambiguous, trivial, or exploitable tasks.
- Contribute tasks and analysis to coding benchmarks and build tools that accelerate task creation and validation.
- Collaborate with research teams to calibrate task difficulty and benchmark coverage.
Requirements
- At least 2 years of professional software engineering experience delivering software systems.
- Very strong Python skills and fluency in another language such as TypeScript, Go, Java, or Rust.
- Hands-on Docker experience and familiarity with reproducible build and test environments.
- Experience navigating unfamiliar, complex codebases and getting them building, running, and tested.
- Working knowledge of coding model training and evaluation, including SWE-bench-style tasks, code evaluations, RLVR, and agent harnesses.
- Experience creating coding tasks from real engineering work and hardening verifiers against flaky tests, unclear requirements, and leaked solutions.
- Experience with frontier AI labs, evaluation teams, RL environments, coding agents, or large enterprise and legacy codebases is helpful.
Benefits
- Remote within the United States or in person in San Francisco, California.