84.51˚

Lead Data Engineer (P3999)

84.51˚
Apply
1 day ago
Chicago, IL, USA or Cincinnati, OH, USAStaff+
H1B sponsor

Base Salary

$125k - $207k/yr

Responsibilities

  • Provide technical leadership and day-to-day direction to a blended onshore/offshore engineering team.
  • Design and develop Databricks-based batch and event-driven data solutions, including streaming ingestion, event processing, and downstream event publishing.
  • Define and maintain event contracts, schemas, versioning, schema evolution, streaming reliability controls, replay, backfill, and dead-letter handling.
  • Develop, maintain, troubleshoot, and optimize data pipelines and models for performance, cost, reliability, and continuity.
  • Implement data quality checks, automated testing, monitoring, error handling, incident response, root cause analysis, and service-level operations.
  • Apply data governance, privacy, security, PII handling, access controls, and lineage practices through Unity Catalog.
  • Build and maintain CI/CD pipelines, infrastructure as code, environment promotion practices, and REST APIs hosted in Kubernetes.
  • Partner with Data Science on data and feature pipelines supporting model development and deployment.
  • Lead design reviews, establish engineering standards, document systems and procedures, and mentor junior engineers.

Requirements

  • Bachelor’s degree in Computer Science, Management Information Systems, Mathematics, Business Analytics, or another technically strong program, or equivalent experience.
  • At least 5 years of professional data engineering experience.
  • At least 5 years of experience developing with Spark, Databricks, or PySpark, and at least 5 years developing with Python and SQL.
  • At least 2 years of experience technically leading or mentoring engineers, including blended onshore/offshore or vendor teams.
  • Experience building event-driven or streaming data solutions with Spark Structured Streaming and a messaging platform such as Kafka or Azure Service Bus.
  • Understanding of event-driven architecture, publish/subscribe, event sourcing, change data capture, delivery guarantees, idempotent processing, windowing, watermarking, and late or out-of-order events.
  • Experience with event schema management using technologies such as Avro, Protobuf, or JSON Schema with a schema registry.
  • Experience with Azure data services including ADLS, Azure Key Vault, Azure Kubernetes Service, and Azure Data Factory.
  • Experience with Git-based workflows, code review, CI/CD, and infrastructure as code.
  • Strong data modeling skills, including dimensional modeling, medallion architecture, Delta Lake, and schema evolution.
  • Experience with Databricks Unity Catalog or equivalent data governance and access-control technology.
  • At least 1 year of experience building REST APIs and at least 1 year of experience working with Kubernetes.
  • Experience with production monitoring, on-call or incident response, and root cause analysis.
  • Strong communication skills, business-to-technical translation, influencing without direct authority, and knowledge of Agile principles.
  • Experience using AI-assisted development tools with disciplined review of generated code.
  • Preferred qualifications include Databricks or Azure certification, legacy ETL or orchestration migration, Databricks Auto Loader, Delta Live Tables or Lakeflow Declarative Pipelines, event-driven API patterns, machine learning workloads or MLOps, and retail, advertising technology, or retail media experience.

Benefits

  • Five-day in-office work schedule.
  • Medical, dental, vision, wellness, and mental health support.
  • 401(k) with Roth option and matching contribution, health savings account with matching contribution, AD&D, and supplemental insurance options.
  • Paid time off including 5 weeks of vacation, 7 health and wellness days, 3 floating holidays, and 6 company-paid holidays annually.
  • Paid maternity, paternity, and family care leave.
  • Eligible for variable compensation in addition to base salary.

Categories

Data Engineering
84.51˚

About 84.51˚

1,001-5,000 employees

84.51° is a retail data science, insights, and media company that builds analytics platforms and retail media solutions for The Kroger Co. and consumer packaged goods brands. It commercializes Kroger first-party loyalty data through products such as 84.51° Insights and Kroger Precision Marketing to support targeting, measurement, and merchandising decisions. Founded in 2015 and headquartered in Cincinnati, it operates as a privately held company offering subscriptions, consulting, and managed media services.

Contact me