Senior Director, NLP & Market Research Data Engineering Lead
Eli Lilly and Company2 days ago
Indianapolis, IN, USAStaff+
Base Salary
$159k - $233k/yr
Responsibilities
- Build and operate enterprise-scale NLP and unstructured-data pipelines on Databricks using medallion architecture.
- Create entity, theme, topic, sentiment, summarization, embedding, and vectorization enrichment layers.
- Own chunking, embedding, vector indexing, retrieval, RAG, and grounding foundations for research agents.
- Make unstructured data governed and AI-ready through data contracts, provenance, PII, consent, classification, and policy enforcement.
- Establish NLP data quality, evaluation, drift, grounding, hallucination checks, monitoring, and alerting.
- Engineer reusable pipelines for social listening, brand health, and secondary Market Research text analysis.
- Set technical standards, review architectures, direct vendor engineering delivery, and mentor engineers and analysts.
- Build the standards and operating model for a future in-house engineering team and grow into direct people leadership.
Requirements
- Bachelor’s degree in Computer Science, Data Engineering, or a related technical discipline.
- 8+ years of data engineering experience, including building and operating production NLP and text-processing pipelines at enterprise scale.
- Experience with lakehouse and medallion architectures using Databricks, Unity Catalog, and Delta.
- Hands-on experience with text parsing and normalization, entity and topic extraction, embeddings, vectorization, retrieval, and RAG.
- Advanced SQL and Python, including PySpark, with production data-quality and testing practices.
- Knowledge of data governance, classification, access control, and sensitive or unstructured data handling.
- Experience with vector databases, search, LLM grounding, and agentic or RAG architectures on enterprise content.
- Familiarity with Market Research data such as social listening, survey verbatims, qualitative transcripts, and secondary or medical-scientific text.
- Experience with ODCS, OPA, Rego, Git-based CI/CD, Azure DevOps or equivalent, dbt, and Terraform.
- Experience in regulated or healthcare data environments and awareness of HIPAA, consent, and MLR or content-governance obligations.
- Ability to influence technical and business partners without direct authority in an ambiguous, matrixed environment.
- Experience mentoring or leading engineers and readiness to build and lead a small engineering team preferred.
Benefits
- Hybrid work arrangement in Indianapolis, Indiana.
- 5–15% travel depending on business need.
- Company bonus eligibility for full-time equivalent employees.
- 401(k), pension, vacation, medical, dental, vision, prescription drug, flexible spending, life insurance, leave, and well-being benefits.
Categories
AI ApplicationsData Engineering
About Eli Lilly and Company
Eli Lilly and Company researches, develops, manufactures, and markets prescription medicines and biologics for patients, sold through healthcare providers, pharmacies, and payers. Core therapeutic areas include diabetes, oncology, immunology, neuroscience, and obesity. Founded in 1876 and headquartered in Indianapolis, it is a public company (NYSE: LLY) with global operations and is building a new advanced manufacturing site for gene therapy and other modalities in Lebanon, Indiana.