Software Engineer- Benchmarking
Office HoursBase Salary
$160k - $210k/yr
Responsibilities
- Prepare, clean, convert, validate, and maintain benchmark datasets and flag ambiguities that could compromise results.
- Build infrastructure and pipelines that run evaluations consistently across model APIs and terminal agents.
- Create lightweight containerized evaluation environments and viewers for tasking and tool-use assessment.
- Support fine-tuning experiments and compare baseline with post-training model performance.
- Build scoreboards and leaderboards showing model-, benchmark-, task-, domain-, and rubric-level results.
- Develop tools to identify failure modes, compare models and agent scaffolds, and track capability changes over time.
- Build HTML viewers and internal tools for task authoring, expert annotation, review, quality control, and structured data collection.
- Collaborate with researchers and engineers to ensure evaluation data and published outputs are accurate, consistent, and integrated.
Requirements
- 4+ years of professional experience building and maintaining complex systems.
- Strong Python engineering skills and experience working in existing codebases and infrastructure.
- Experience preparing, cleaning, validating, and maintaining datasets.
- Experience with Docker and reproducible execution environments.
- Ability to collaborate with researchers and scientists and translate methodology into working systems.
- Hands-on AI evaluation experience or experience with Harbor, Terminal-Bench, or Inspect is a strong plus.
- Experience fine-tuning or post-training open-source LLMs, agentic or tool-use evaluation, LLM-as-judge or rubric-based grading, data-heavy dashboards, leaderboards, visualizations, or related open-source or published benchmark work is preferred.
Benefits
- Medical, dental, and vision coverage.
- 401(k).
- Monthly wellness and fitness stipend.
- Paid time off and company holidays.
- Annual company off-sites in Tahoe, Mendocino, Mexico City, San Diego, and Park City.
- Parent-friendly policies, paid family leave, and remote flexibility.
- Fully remote or hybrid work from the San Francisco or New York City offices.
Categories
About Office Hours
Office Hours is an on-demand expert network connecting leading organizations with the expertise they need, across industries, functions, and hard-to-reach domains. We believe human knowledge is the world's most valuable asset. Yet, even in an increasingly connected world, most of what people know remains hard to reach and underutilized. Office Hours is on a mission to change that. We're building toward a future where human expertise is easily accessible and infinitely scalable, where people realize the true value of what they know and turn it into real income and impact. Experts earn by sharing what they know through live calls, surveys, or AI interviews. Their knowledge unlocks key insights, drives mission-critical strategies, and solves real problems for clients. Trust sits at the center of it all: every expert is carefully vetted, so clients know they're speaking with exactly who they expect, and experts know every opportunity is real. Our platform and team handle everything behind the scenes, from matching to payments, so experts and clients can focus on their interaction, not the logistics. Office Hours is a hyper-growth startup headquartered in San Francisco, with growing teams in New York and Bangalore. Our customers include leading technology, life sciences, investment, consulting and AI organizations. We're backed by the top marketplace investors and operators behind companies like DoorDash, Airbnb, and Affirm.