5 months ago
Remote, United StatesSenior
Responsibilities
- Own the roadmap, priorities, platform standards, architecture reviews, and shared clinical data products supporting the AI portfolio.
- Build batch and streaming pipelines that ingest Epic SDE, ADT feeds, lab results, and other clinical data into Databricks.
- Curate reusable clinical feature tables for AI model training, validation, and monitoring.
- Design and operate RAG document ingestion, normalization, chunking, embedding, vector database, and retrieval pipelines.
- Administer Databricks workspaces, Unity Catalog, clusters, compute, user and group access, cost monitoring, and Feature Store.
- Implement hybrid search, reranking, relevance filtering, data quality gates, and retrieval quality monitoring.
- Provide formal input on performance reviews and comply with organizational policies and job-specific obligations.
Requirements
- Bachelor’s degree in a related field is required; a master’s degree in a related field is preferred.
- At least 5 years of relevant data engineering experience, with experience building batch and streaming data pipelines.
- Expert-level experience with Databricks, Delta Live Tables, PySpark, Unity Catalog, and Feature Store.
- Hands-on real-time data ingestion experience with Kafka, Spark Structured Streaming, or comparable frameworks.
- Strong SQL and Python skills, including pandas and PySpark, for data transformation and feature engineering.
- Experience administering Databricks workspaces, including cluster policies, compute management, access controls, and cost monitoring.
- Familiarity with clinical data models and healthcare sources such as EHR extracts, ADT feeds, lab results, and claims data is strongly preferred.
- Experience with Epic data extraction methods including SDE, FHIR, and epic-ws is a significant plus.
- Understanding of data governance principles including lineage, quality monitoring, and access controls.
Benefits
- Work from home within Pennsylvania, with a regular full-time schedule of 40 hours per week on day shift.
- Healthcare benefits, including vision, dental, and domestic partner coverage, are available from day one for full-time and part-time positions.
- Collaborative, collegial, diverse, affirmative-action and equal-opportunity workplace.
Tech Stack
Categories
AI ApplicationsData Engineering
