Lead Digital Data Engineer
BJ's Wholesale Club15 days ago
Marlborough, MA, USAStaff+
Base Salary
$129k - $164k/yr
Responsibilities
- Design and maintain BJ’s scalable Databricks Lakehouse using Delta Lake and Bronze/Silver/Gold medallion architecture.
- Build batch and real-time data pipelines integrating Kafka, DynamoDB, S3, and other AWS services with Databricks.
- Integrate Adobe Analytics, AEP, Google Analytics, transactional data, and operational data.
- Implement automated data quality validation, alerting, service-level agreements, and pipeline health monitoring.
- Lead data governance through Unity Catalog, including access control, lineage, and classification.
- Develop Genie Spaces and AI/BI dashboards with certified metrics for business self-service.
- Productionize Databricks agents and AI use cases using Mosaic AI, MLflow, Model Serving, and Vector Search.
- Optimize performance and cost across clusters, SQL Warehouses, and Delta tables.
- Establish CI/CD, version control, and automated testing for data pipelines using Databricks Asset Bundles, Git workflows, and Terraform.
- Own and prioritize the digital data and analytics roadmap with product leaders.
- Lead and mentor a distributed team of up to 10 engineers and provide status and risk updates to senior leadership.
Requirements
- Bachelor’s degree in Computer Science, Information Technology, or equivalent, plus 7+ years of data engineering experience.
- At least 3 years of hands-on Databricks experience in production.
- Expertise in Apache Spark, PySpark, SQL, Delta Lake, and Lakehouse architecture.
- Experience with Unity Catalog, Databricks Workflows, and SQL Warehouses.
- Experience integrating AWS S3, Kafka including MSK or Confluent, DynamoDB, Lambda, and IAM.
- Experience building data quality and validation frameworks.
- Experience with Genie Spaces, AI/BI dashboards, and Databricks agents or generative AI solutions including Mosaic AI and MLflow.
- Experience building large-scale distributed data systems on AWS, Azure, or GCP.
- Experience working in an Agile Scrum environment and leading distributed teams.
- Experience with SDLC, CI/CD, version control, and test automation for data pipelines.
- Familiarity with AI tools including ChatGPT is valued.
Benefits
- Weekly pay and access to earned pay through Daily Pay, with a fee possibly applying.
- Complimentary BJ’s Club Card Membership and a free Supplemental Membership for a household member.
- Paid vacation, personal, sick, holiday, bereavement, and jury duty leave.
- Medical, dental, vision, HSA, and flexible spending account options.
- 401(k) retirement savings plan with company match for eligible team members.
- Employee Stock Purchase Plan with a 15% discount on BJ’s common stock.
- Hybrid schedule with in-office work Tuesday through Thursday at the BJ’s Club Support Center in Marlborough, Massachusetts, and remote work Monday and Friday.
Tech Stack
Categories
Data Engineering
About BJ's Wholesale Club
BJ’s Wholesale Club operates membership-based warehouse clubs and an e-commerce site selling groceries, household goods, electronics, and fuel to families and small businesses. Revenue comes from annual memberships and retail sales, including private-label brands like Wellsley Farms and Berkley Jensen. Founded in 1984 and headquartered in Marlborough, Massachusetts, BJ’s is a public company listed on the NYSE.