Lead Research Engineer – AI Evaluation & Benchmarking
Thomson Reuters2 days ago
Base Salary
$127k - $273k/yr
Responsibilities
- Lead technical delivery and engineering planning, including prioritization, project staging, estimation, staffing recommendations, dependency management, and adapting plans to available resources.
- Design and build scalable evaluation infrastructure, production-quality Python software, APIs, reusable frameworks, and automated pipelines for AI model benchmarking.
- Translate research methods into working software by connecting model APIs, test datasets, scoring methods, and human assessment workflows.
- Maintain reproducible model comparisons and measure model quality, operating cost, latency, and evaluation reliability.
- Build interfaces, documentation, evaluation reports, and diagnostic tools that help researchers and product teams run evaluations and interpret failures.
- Integrate automated testing, deployment, monitoring, performance optimization, access controls, permitted-use rules, and data-retention requirements.
- Collaborate across research, data engineering, platform engineering, and product teams; mentor engineers and guide architecture and delivery decisions.
Requirements
- 8–10+ years of relevant software engineering experience building and maintaining production systems involving machine learning or complex data.
- Strong hands-on Python skills, including API design, automated testing, and processing structured and unstructured data.
- Experience with technical leadership, architecture decisions, engineering planning, staffing and effort estimation, project execution, and changing priorities.
- Experience collaborating with research scientists or machine learning practitioners to translate experiments and research methods into software.
- Experience integrating or testing machine learning models, interpreting evaluation results, understanding language models, and identifying unreliable comparisons or test-data leakage.
- Cloud engineering experience with AWS, Azure, or a comparable platform, including distributed workflows, automated deployment, and monitoring.
- Working knowledge of SQL and experience developing software that processes structured and unstructured data.
- Strong collaboration, communication, problem-solving, mentoring, and cross-functional technical delivery skills.
- Bachelor’s degree in Computer Science, Engineering, or a related field, or equivalent practical experience.
- Preferred qualifications include LLM evaluation and benchmarking pipelines, automated scoring, human assessment, experiment tracking, dataset versioning, reproducible evaluation, natural language processing, information retrieval, model cost optimization, reusable libraries or frameworks, and research or open-source contributions.
Benefits
- Flexible hybrid work model for office-based roles.
- Flexible work arrangements, including work from anywhere for up to 8 weeks per year.
- Career development, continuous learning, skills development, and leadership-growth programs.
- Comprehensive benefits including flexible vacation, mental health days, Headspace access, retirement savings, tuition reimbursement, incentive programs, and wellbeing resources.
- US benefits include health, dental, vision, disability, life insurance, 401(k) matching, paid holidays, parental leave, sabbatical leave, commuter benefits, adoption and surrogacy assistance, and employee stock purchase plan.
- Two paid volunteer days annually and opportunities for pro-bono consulting and ESG initiatives.
- Eligible locations include New York City, San Francisco, Los Angeles, Irvine, McLean, Washington, DC, and Ontario, Canada; the posting is for current or future vacancies.
About Thomson Reuters
Thomson Reuters builds research platforms, workflow software, and data services for legal, tax and accounting, compliance, and government professionals, and operates the Reuters global news service. Flagship products include Westlaw for legal research and ONESOURCE and Checkpoint for tax and accounting, sold primarily via subscriptions and enterprise licenses. A public company headquartered in Toronto, it was formed in 2008 by combining Thomson Corporation and Reuters Group and is listed on the TSX and Nasdaq as TRI.