
Associate Director – RWD Data Engineering
Eli Lilly and Company2 days ago
Indianapolis, IN, USAStaff+
Base Salary
$137k - $200k/yr
Responsibilities
- Design, develop, deploy, and optimize cloud-native Databricks data products and high-throughput pipelines for large-scale claims and EHR data.
- Use Python, SQL, R, PySpark, Delta Lake, Databricks Workflows, and related tools to build reproducible ETL/ELT transformations and analysis-ready healthcare datasets.
- Implement data quality, validation, reconciliation, regression testing, monitoring, and alerting frameworks for every data product.
- Apply healthcare coding standards and OMOP CDM methods to produce domain datasets, code lists, and analysis-ready data assets.
- Optimize distributed SQL and Spark workloads across datasets ranging from tens of millions to billions of records.
- Build interactive analytic applications using Dash, R Shiny, Streamlit, Databricks Apps, and Databricks SQL.
- Build and evaluate AI-enabled solutions using frontier model APIs, Databricks Model Serving, retrieval, agents, and governed data products.
- Apply Git, pull requests, code review, automated testing, CI/CD, and Databricks Asset Bundles throughout the development lifecycle.
- Serve as a technical point of contact for assigned RWD sources and products and advise scientific and technical collaborators on appropriate use and limitations.
- Contribute to engineering standards, transformation templates, code-list repositories, documentation, training materials, and an inclusive quality-focused culture.
Requirements
- Bachelor’s degree in Computer Science, Applied Mathematics, Engineering, Statistics, Information Technology, or Bioinformatics.
- At least 5 years of hands-on data engineering experience focused on healthcare or life sciences real-world data, including administrative claims and/or EHR datasets.
- At least 1 year of production Databricks development using Python/PySpark, SQL, and Delta Lake on real-world healthcare data at scale.
- At least 1 year of experience designing and implementing data quality and validation checks for large clinical datasets.
- At least 1 year of experience using AI coding assistants to develop production software and building at least one solution with frontier AI model APIs.
- Preferred qualifications include a master’s degree, OMOP CDM and Athena experience, distributed computing experience, analytic application development, healthcare coding and RWD vendor knowledge, advanced SQL optimization, pipeline observability, FDA/EMA RWD guidance familiarity, Databricks certification, MCP experience, and a code or application portfolio.
- Experience with claims and EHR standards and structures including ICD-10, ICD-9, NDC, RxNorm, SNOMED CT, CPT, HCPCS, LOINC, ATC, HL7 FHIR, and OMOP is preferred.
- Experience with Git-based version control, large-scale ingestion pipelines, data quality tooling, and cross-functional collaboration with technical and scientific partners is preferred.
Benefits
- Full-time employees may be eligible for a company bonus based on company and individual performance.
- Benefits include a company-sponsored 401(k), pension, vacation, medical, dental, vision, prescription drug, flexible spending, life insurance, leave, employee assistance, fitness, and employee club benefits.
- The anticipated base salary is $136,500-$200,200.
Tech Stack
Amazon RedshiftApache AirflowApache SparkAWSAzureDatabricksdbtDockerGitGitHub ActionsGoogle BigQueryGoogle Cloud PlatformMLflowPandasPythonRR ShinySnowflakeSQL
Categories
Data Engineering
About Eli Lilly and Company
Eli Lilly and Company researches, develops, manufactures, and markets prescription medicines and biologics for patients, sold through healthcare providers, pharmacies, and payers. Core therapeutic areas include diabetes, oncology, immunology, neuroscience, and obesity. Founded in 1876 and headquartered in Indianapolis, it is a public company (NYSE: LLY) with global operations and is building a new advanced manufacturing site for gene therapy and other modalities in Lebanon, Indiana.