Evaluation Engineer
Freshfields Bruckhaus Deringer4 days ago
London, United Kingdom or New York, NY, USASenior
Base Salary
$100k - $200k/yr
Responsibilities
- Implement automated evaluation cases and supporting scripts using established frameworks and senior-engineer guidance.
- Prepare and maintain evaluation datasets, source documents, reference examples, regression suites, and approved production-failure test cases.
- Run functional and integration tests covering document ingestion, tool calls, structured outputs, error handling, accuracy, completeness, citations, retrieval relevance, instruction following, and escalation.
- Review outputs and execution traces, reproduce issues, document evidence, and distinguish data, retrieval, model, and application defects.
- Compare results across model, prompt, and application versions and report changes, uncertainty, and limitations accurately.
- Support calibration of automated graders by comparing their scores with human reviews and investigating inconsistencies.
- Contribute evaluations to CI/CD integration and follow confidentiality, permitted-data-use, and version-control requirements.
Requirements
- At least five years of software development experience through employment, placements, or substantial projects.
- Programming experience with languages such as Python, Java, or C#.
- Foundational understanding of large language models, including capabilities, limitations, and common failure modes.
- Ability to use coding agents with appropriate prompting, context management, validation, security, responsible-use, and human-review practices.
- Working knowledge of APIs, Git, automated testing, software development lifecycle practices, and CI/CD tools such as Jira and Jenkins.
- Foundational understanding of statistics and measurement, including sampling, precision, and recall, plus strong analytical and documentation skills.
- Practical AI application evaluation experience is preferred but not required; relevant academic, internship, or project experience may be considered.
About Freshfields Bruckhaus Deringer
Freshfields Bruckhaus Deringer is a global law firm advising corporations, financial institutions, and governments on complex transactions, disputes, and regulatory matters across jurisdictions. Structured as a partnership headquartered in London and founded in 1743, it operates offices across Europe, Asia, the Middle East, and the United States. Its fee-based practice spans M&A, antitrust, arbitration and litigation, finance and capital markets, and compliance and investigations.