1 day ago
Base Salary
$230k - $340k/yr
Responsibilities
- Build synthetic data pipelines that generate, filter, and curate training and evaluation samples for legal reasoning and agentic workflows.
- Design evaluations and verifiers, calibrate them against expert judgment, and investigate their failure modes.
- Build and maintain capability benchmarks such as the Legal Agent Benchmark and product-focused evaluations for customer workflows.
- Work with legal domain experts to define quality metrics, rubrics, and grading mechanisms.
- Own training and evaluation data quality, including provenance, coverage, and contamination safeguards.
- Connect product usage and customer feedback to new evaluations and training data.
- Investigate model failures and use findings to guide training and release decisions.
Requirements
- Experience building data pipelines, ML evaluation systems, or experimentation tooling.
- Strong Python skills and experience owning production-quality code.
- Rigorous understanding of measurement, statistical reasoning, experimental design, and misleading metrics.
- Ability to translate ambiguous goals and expert judgment into structured datasets, rubrics, and measurable criteria.
- Experience with synthetic data, LLM evaluations, automated raters, or preference data is preferred.
- Strong data and measurement engineering experience close to ML model development is valuable even without prior LLM experience.
Benefits
- Full-time role with compensation of $230,000-$450,000 USD depending on location.
Tech Stack
Categories
About Harvey
Harvey builds domain-specific generative AI for legal and other professional services, delivered as an enterprise platform and APIs to automate contract analysis, due diligence, compliance, and litigation workflows. Founded in 2022 and headquartered in San Francisco, it sells to law firms and corporate legal departments on enterprise agreements; investors include Sequoia Capital and OpenAI. Customers include multiple Am Law 100 firms and Fortune 500 companies’ legal teams.