3 hours ago
London, United KingdomSenior
Responsibilities
- Design greenfield product evaluations, including endpoints, comparators, ground truth, sample size, and analysis plans.
- Build, optimize, and validate LLM judges and scoring rubrics against clinician ground truth.
- Execute evaluations, perform statistical analyses, and prepare reports for product, regulatory, and academic use.
- Conduct post-market surveillance of clinical AI output quality and interpret clinical safety findings for product changes.
- Implement prompt, rubric, evaluation-harness, and production code changes in Python or TypeScript.
- Contribute to peer-reviewed publications and provide clinical judgment on nuanced or incorrect model outputs.
Requirements
- Medical degree and clinical practice involving independent clinical judgment.
- Advanced knowledge of study design, inferential statistics, agreement and reliability measures, sample size, and multiple-comparison handling.
- Thesis- or publication-level academic research experience.
- Experience evaluating medical software or clinical AI, including building or validating LLM-as-judge pipelines.
- Ability to read and modify production code in Python or TypeScript and complete normal code review.
- Ability to work independently with light direction and communicate clearly through evaluation reports, surveillance summaries, and manuscripts.
- Preferred qualifications include medical-device post-market surveillance under TGA, MHRA, FDA, or EU MDR frameworks; peer-reviewed clinical AI or LLM evaluation publications; machine learning beyond prompting; relevant specialty expertise; or an academic or clinical-site study network.
Benefits
- Equity from day one.
- Comprehensive private medical and dental coverage through Bupa, plus mental health, coaching, and wellbeing support through Sonder.
- £100 per month Healthy Heidi’s stipend.
- Paid parental leave of 26 weeks for primary carers and 18 weeks for secondary carers, subject to eligibility.
- £7,000 fertility-support payment, subject to eligibility.
- £700 annual learning and development budget.
- £500 one-off home-office setup budget.
- Recharge days after major milestones and busy periods.
- Work from anywhere for up to four weeks per year.
- Ten days of clinical leave per year for eligible clinical roles.
- Hybrid work environment with three days in the office.
Tech Stack
Categories
About Heidi
Heidi builds an AI care partner for clinicians that automates documentation and workflows, including an AI medical scribe, Evidence for point‑of‑care research, and Comms for patient coordination. It sells a freemium and enterprise SaaS platform to healthcare providers and health systems. Headquartered in Melbourne and privately held, Heidi reports supporting over 2.7 million patient interactions each week in 110 languages across 190 countries.