2 months ago
London, United KingdomSenior
Responsibilities
- Design and build healthcare-domain evaluation systems for LLM capabilities and interpret and communicate their results.
- Build evaluation pipelines, curate and synthesize datasets, and develop automated analyses.
- Run experiments on prompting techniques using internal and industry benchmarks.
- Improve internal tooling for implementing, running, and analyzing evaluations.
- Design benchmarking and regression-testing capabilities for model accuracy, safety, and utility.
- Collaborate cross-functionally with research, product, design, and engineering teams and contribute to product roadmaps.
Requirements
- Bachelor’s or higher degree in computer science or a related technical discipline.
- Significant Python programming experience or machine learning research experience.
- Experience building with and around LLMs, including prompt, context, and harness engineering.
- Deep expertise evaluating and understanding agent performance.
- Experience collaborating across functions and delivering high-quality results amid ambiguity.
- Experience taking products or capabilities from zero to one with a bias toward shipping and learning.
- Ability to foster knowledge sharing and contribute to a positive, inclusive work environment.
- Preferred: healthcare technology or health-domain experience.
- Preferred: data engineering experience involving text dataset sourcing, curation, and processing at scale.
- Preferred: strong written and verbal communication skills and ability to work with product managers, designers, and engineers.
- Preferred: interest in learning emerging AI technologies, patterns, and best practices.
Benefits
- The role is based in the United Kingdom and is open for a minimum of five days, with applications accepted until the position is filled.
- Certain roles may be eligible for benefits and other compensation; additional benefits and pay information is provided through Microsoft’s corporate pay resources.
Tech Stack
Categories
About Microsoft
Microsoft develops operating systems, productivity software, cloud services, developer tools, and consumer devices for individuals, enterprises, and governments. Its main products include Windows, Microsoft 365, Azure, Visual Studio/GitHub, Xbox, and LinkedIn; revenue comes from software subscriptions and licenses, cloud consumption, hardware sales, and advertising. Founded in 1975 and headquartered in Redmond, Washington, Microsoft is a public company traded on Nasdaq.