2 months ago
Base Salary
$150k - $180k/yr
Responsibilities
- Design and build production data pipelines with Lakeflow Declarative Pipelines, Auto Loader, Structured Streaming, data quality expectations, and Declarative Automation Bundles deployments.
- Architect and implement Databricks Lakehouse solutions using medallion architecture, Delta Lake, and Unity Catalog.
- Build and maintain transformation layers with DLT pipelines, PySpark notebooks, and dbt.
- Develop governed data and AI foundations using Unity Catalog, Feature Store, MLflow, Model Serving, Lakebase, and Databricks Apps.
- Collaborate with product and backend engineers on data models, APIs, application data contracts, and data-to-application architectures.
- Consult with clients on data challenges, strategy, architecture, and sustainable implementation approaches.
- Design data governance practices covering access control, lineage, data quality policies, and compliance.
- Work across multi-cloud environments, primarily AWS and Azure, and help build Livefront’s Databricks practice through accelerators, enablement, certifications, and partner go-to-market materials.
Requirements
- 7–10 years of data engineering experience, including at least 5 years in production Databricks environments, preferably in consulting or client delivery.
- Working knowledge of AWS and Azure services relevant to Databricks deployments, with GCP familiarity preferred.
- Deep production experience with Lakeflow Declarative Pipelines, Auto Loader, Structured Streaming, Lakeflow Jobs, and Unity Catalog, including fine-grained access control and lineage.
- Experience designing production-scale Lakehouse architectures, including medallion patterns, Delta Lake table design, partitioning, Z-ordering, and query optimization.
- Hands-on experience with data pipeline testing, observability, data quality frameworks, Git-based version control, and CI/CD deployments using Declarative Automation Bundles.
- Strong proficiency in SQL and Python and understanding of data modeling, schema design, and query optimization.
- Strong communication, problem-solving, organization, peer-review, and stakeholder collaboration skills.
- Preferred qualifications include experience with Kafka, Kinesis, alternative cloud data platforms, healthcare or fintech, MLOps or LLMOps, MLflow, Model Serving, Vector Search, Databricks Apps, Lakebase, Java, Go, Scala, technical diagramming, public education, open-source data projects, or relevant Databricks and Apache Spark certifications.
Tech Stack
Categories
Data Engineering
About Livefront
Livefront is a digital product consultancy that designs and builds mobile, web, and AI-enabled experiences for enterprises and startups. The privately held firm, founded in 2001 and headquartered in Minneapolis, provides strategy, design, and software engineering services on a client-services model. Notable clients have included CVS, Samsung, General Mills, Optum, and startups such as HomeSpotter and Credly.