Software Engineer- Benchmarking
Office Hours2 months ago
Remote, United StatesMid Level
Base Salary
$160k - $210k/yr
Responsibilities
- Prepare, clean, convert, validate, and maintain benchmark datasets and flag ambiguities that could compromise results.
- Build infrastructure and pipelines that run evaluations consistently across model APIs and terminal agents.
- Create lightweight containerized evaluation environments and viewers for tasking and tool-use assessment.
- Support fine-tuning experiments and compare baseline with post-training model performance.
- Build scoreboards and leaderboards showing model-, benchmark-, task-, domain-, and rubric-level results.
- Develop tools to identify failure modes, compare models and agent scaffolds, and track capability changes over time.
- Build HTML viewers and internal tools for task authoring, expert annotation, review, quality control, and structured data collection.
- Collaborate with researchers and engineers to ensure evaluation data and published outputs are accurate, consistent, and integrated.
Requirements
- 4+ years of professional experience building and maintaining complex systems.
- Strong Python engineering skills and experience working in existing codebases and infrastructure.
- Experience preparing, cleaning, validating, and maintaining datasets.
- Experience with Docker and reproducible execution environments.
- Ability to collaborate with researchers and scientists and translate methodology into working systems.
- Hands-on AI evaluation experience or experience with Harbor, Terminal-Bench, or Inspect is a strong plus.
- Experience fine-tuning or post-training open-source LLMs, agentic or tool-use evaluation, LLM-as-judge or rubric-based grading, data-heavy dashboards, leaderboards, visualizations, or related open-source or published benchmark work is preferred.
Benefits
- Medical, dental, and vision coverage.
- 401(k).
- Monthly wellness and fitness stipend.
- Paid time off and company holidays.
- Annual company off-sites in Tahoe, Mendocino, Mexico City, San Diego, and Park City.
- Parent-friendly policies, paid family leave, and remote flexibility.
- Fully remote or hybrid work from the San Francisco or New York City offices.
Categories
About Office Hours
Office Hours builds an on-demand expert network and workflow platform connecting investment, consulting, and operating teams with vetted specialists across technology, software, and healthcare. Clients book live calls, surveys, and AI-assisted interviews while the platform handles sourcing, compliance, scheduling, and payments; experts earn by advising and training AI models. Founded in 2020 and headquartered in San Francisco, the privately held company operates a marketplace model, charging clients for expert consultations and projects.