LillyDirect Pharmacy Data Engineer
Eli Lilly and Company1 day ago
Indianapolis, IN, USASenior
Base Salary
$66k - $158k/yr
Responsibilities
- Design, build, and maintain scalable batch and near-real-time data pipelines using Python and PySpark on Databricks and AWS.
- Develop and optimize data models, ETL/ELT workflows, and integrations across systems using Delta Lake and lakehouse patterns.
- Support data loads, extracts, and reconciliation for Reltio and Transcend integrations.
- Implement data quality, lineage, reconciliation, governance, security, access controls, masking, and retention for profile, consent, PII, and PHI data.
- Collaborate with technical leads, product owners, architects, analysts, vendors, and business stakeholders to translate requirements into technical solutions.
- Own pipelines and workstreams from design through production and contribute to architectural and design decisions.
- Promote coding, code review, automated testing, and deployment best practices through CI/CD pipelines.
- Monitor, troubleshoot, and improve pipeline performance, reliability, and cost while supporting production incidents and root-cause analysis.
- Automate manual processes, optimize data workflows, mentor junior engineers, and share data engineering and cloud expertise.
Requirements
- Bachelor’s degree in Computer Science, Information Technology, or Management Information Systems.
- At least 5 years of data engineering experience using SQL, Python, PySpark, and cloud platforms such as AWS, Azure, or GCP.
- Experience with cloud services including Lambda, Glue, EMR, S3, Redshift, Athena, and IAM roles and policies.
- At least 2 years working in Agile environments with hands-on GitHub and CI/CD deployment experience.
- Hands-on Databricks experience, including workspace configuration, notebook development, Delta Lake, and performance optimization.
- At least 1 year of experience with Airflow or Databricks Workflows for workflow automation.
- Experience with scalable data pipelines, data lakehouse, data warehouse, data mart, ETL/ELT, data modeling, large datasets, and secure data governance.
- Knowledge of regulated data practices, including PII/PHI handling, privacy requirements such as HIPAA, data quality, and consent or preference management.
- Experience with Reltio or a comparable MDM platform and familiarity with Transcend or a similar consent platform.
- Familiarity with Kafka, Kinesis, or MSK; Unity Catalog; Terraform; and data quality frameworks such as Great Expectations or dbt tests.
- Strong collaboration, communication, problem-solving, attention to detail, and ability to work with technical and non-technical stakeholders.
- Pharmaceutical or healthcare industry experience and AWS Certified Data Engineer or Databricks Certified Data Engineer credentials are preferred.
- Must be authorized to work full-time in the United States without employer support or sponsorship for work authorization or visas.
Benefits
- Full-time employees may be eligible for a company bonus based partly on company and individual performance.
- Benefits may include a 401(k), pension, vacation, medical, dental, vision, prescription drug, flexible spending, life insurance, leave, well-being, fitness, employee assistance, and employee club benefits.
Tech Stack
Amazon RedshiftApache AirflowApache KafkaAWSAzureDatabricksdbtGoogle Cloud PlatformPythonSQLTerraform
Categories
Data Engineering
About Eli Lilly and Company
Eli Lilly and Company researches, develops, manufactures, and markets prescription medicines and biologics for patients, sold through healthcare providers, pharmacies, and payers. Core therapeutic areas include diabetes, oncology, immunology, neuroscience, and obesity. Founded in 1876 and headquartered in Indianapolis, it is a public company (NYSE: LLY) with global operations and is building a new advanced manufacturing site for gene therapy and other modalities in Lebanon, Indiana.