Research Engineer - Data and Evaluation
Prime Intellect7 hours ago
Responsibilities
- Design and build challenging evaluations for coding and long-horizon agents.
- Build executable environments in Verifiers with realistic task setup and clear success criteria.
- Develop and test scoring logic and verifiers while investigating reward hacks and misleading progress measures.
- Trace model and harness failures to distinguish model limitations from environment or scoring problems.
- Build and scale RL data pipelines that generate and curate coding and agentic tasks.
- Partner with RL researchers to evaluate task and training-data changes on held-out tasks.
Requirements
- Strong Python software engineering skills and experience building data pipelines, evaluation systems, or ML workflows used by others.
- Hands-on experience with language models or agents and understanding of how training data and evaluation choices affect model behavior.
- Ability to analyze complex data and model outputs, identify failure modes, and turn ambiguous quality questions into testable measurements.
- Experience owning technical projects through experimentation, implementation, and validation while collaborating with researchers and engineers.
- Hands-on experience building technical evaluations or agent environments, with the ability to inspect model traces and debug harness and scoring code.
- Preferred experience includes synthetic data generation, dataset curation, distributed processing, evaluation methodology, executable coding tasks, agent environments, verifiers, reward design, experiment analysis, statistical testing, contamination detection, reproducible research tooling, RL or post-training data, model-based grading, long-horizon coding agents, or reward-exploitation diagnosis.
Benefits
- Visa sponsorship and relocation assistance are available for international candidates.
- Flexible work arrangements with remote or in-person work at the San Francisco offices.
- Quarterly team off-sites, hackathons, conferences, and learning opportunities.
- The posting states a cash compensation range of $150-350k, including equity incentives.
Tech Stack
Categories
About Prime Intellect
Prime Intellect builds a full-stack platform for training frontier AI models, giving enterprises access to distributed compute and agentic training infrastructure to develop their own systems. The company operates as a platform and open research lab, with customers leveraging its infrastructure rather than building it in-house. Headquartered in San Francisco and privately held, it has raised a Series A round backed by investors including Founders Fund and NVIDIA.