Associate Director, Clinical AI Evaluation and Responsible Deployment
AstraZeneca1 day ago
Barcelona, SpainStaff+
Responsibilities
- Define and implement evaluation strategies for clinical AI models and agents across development, release, monitoring, and change control.
- Build evaluation harnesses, curated test sets, simulation environments, automated regression suites, and analysis pipelines using Python and related technologies.
- Translate clinical expert judgment into task definitions, scoring rubrics, error taxonomies, and clinically meaningful acceptance thresholds.
- Evaluate complete AI workflows, including retrieval quality, tool selection, orchestration, source fidelity, abstention, human handoffs, latency, and downstream operational impact.
- Design evaluation methods for noisy, sparse, delayed, or expert-dependent ground truth, including adjudication, inter-rater agreement, challenge sets, prospective studies, and post-deployment surveillance.
- Lead failure analysis and red-teaming for hallucination, unsupported claims, automation bias, data leakage, prompt injection, subgroup performance, and unsafe workflow behavior.
- Establish traceability from intended use and user needs through requirements, test evidence, release decisions, monitoring, and revalidation.
- Align evaluation evidence with GCP, GxP, data-integrity, validation, audit, and inspection-readiness expectations.
- Advise teams on progression from prototype to controlled pilot, broader deployment, or regulated use based on available evidence.
- Create reusable evaluation components and standards for agentic workflows across the Clinical Development AI portfolio.
- Communicate findings, uncertainty, and residual risks to technical, clinical, quality, regulatory, and executive audiences.
- Mentor scientists and engineers in experimentation, reproducibility, and responsible clinical AI development.
Requirements
- PhD in Machine Learning, Computer Science, Statistics, Biostatistics, Biomedical Informatics, Computational Biology, or a related quantitative discipline; alternatively, an MD, master’s degree, or equivalent experience with a strong computational record.
- At least 4 years of post-PhD or equivalent experience evaluating, validating, or deploying AI/ML systems in healthcare, life sciences, clinical research, or another safety- or evidence-critical environment.
- Current hands-on coding ability in Python and SQL, including automated evaluation pipelines, large-scale data analysis, testing, and version-controlled development.
- Strong grounding in experimental design, statistical inference, uncertainty, error analysis, and measurement reliability.
- Experience evaluating LLMs or agentic systems, including retrieval, tool use, structured outputs, multi-step workflows, robustness, and human-in-the-loop performance.
- Experience developing evaluation approaches for noisy labels, differing expert opinions, or delayed outcomes.
- Experience producing audit-ready evidence under GCP and GxP, including model-risk management, audit trails, and inspection readiness.
- Ability to work with clinical stakeholders to define intended use, harmful failure modes, decision thresholds, and acceptable human oversight.
- Deep understanding of trustworthy AI principles and lifecycle governance, with the ability to convert them into release and monitoring criteria.
- Excellent communication skills and judgment when explaining technical evidence and residual risk to clinical, quality, regulatory, and executive audiences.
- Experience leading cross-functional technical work through influence and making evidence-based go or no-go recommendations under ambiguity.
- Preferred qualifications include clinical trial conduct, medical monitoring, pharmacovigilance, or clinical data management experience; prospective or human-factors evaluation experience; familiarity with causal inference, calibration, subgroup analysis, weak supervision, or active learning; experience with LLM evaluation platforms, observability, MLOps/LLMOps, reproducible experimentation, or production monitoring; and relevant publications, standards work, open-source contributions, or regulator and industry-consortium engagement.
Benefits
- The role requires working from the office an average minimum of three days per week, with individual flexibility.
- AstraZeneca offers an inclusive and diverse working environment and considers applications from qualified candidates regardless of their characteristics.
Categories
About AstraZeneca
AstraZeneca is a global biopharmaceutical company that discovers, develops, manufactures, and commercializes prescription medicines for patients and healthcare systems. Its portfolio spans oncology, cardiovascular, renal and metabolism, respiratory and immunology, and rare diseases (expanded by the 2021 acquisition of Alexion). Founded in 1999 and headquartered in Cambridge, UK, it is a public company listed on the London Stock Exchange and Nasdaq, generating revenue primarily from drug sales and partnerships.