Data Engineer Principal_3003
Allianz SE3 months ago
Minneapolis, MN, USAStaff+
Base Salary
$110k - $150k/yr
Responsibilities
- Assess, reverse-engineer, document, and validate an existing enterprise data platform from source ingestion through data consumption.
- Identify and remediate platform gaps, defects, technical debt, and incorrect or suboptimal implementations.
- Validate CDC, slowly changing dimensions, deduplication, business reconciliation, audit, quality, and governance controls.
- Design and implement modern lakehouse patterns while preserving feature parity and business logic during migration.
- Re-implement ingestion, transformation, orchestration, audit, reconciliation, and data quality pipelines in Databricks.
- Support parallel-run validation, output consistency checks, cutover activities, and legacy component decommissioning.
- Collaborate with business, data governance, and architecture stakeholders to validate business rules and data quality requirements.
- Mentor less experienced engineers and review code and technical designs.
Requirements
- 7+ years of hands-on data engineering experience, including platform migration or re-engineering work.
- Hands-on expertise with Azure Synapse Analytics, Azure Data Lake Storage Gen2, Delta Lake, Azure Databricks, Apache Spark, PySpark, and Spark SQL.
- Strong Python and SQL programming skills and experience with ETL/ELT, data modeling, data integration, and large-scale data platforms.
- Experience with SCD Types 0/1/2, CDC, deletion processing, reconciliation, audit frameworks, data quality controls, and enterprise governance.
- Experience migrating legacy data warehouse or Synapse workloads to a Databricks lakehouse and re-implementing governance frameworks.
- Experience integrating Oracle DB, SQL Server, Azure SQL Database, file systems, Salesforce, and APIs.
- Experience with Azure DevOps or GitHub Actions, Terraform or ARM, Docker, and automated data pipeline testing.
- Proven ability to analyze complex existing implementations, distinguish business intent from defects, and make sound re-engineering decisions.
- Experience working in regulated or enterprise-scale environments is preferred; financial services experience is a plus.
- Experience with Unity Catalog, Azure Purview, streaming platforms, GenAI or ML platform integration, observability tools, and BI tools is preferred.
- Strong communication skills and ability to work with business, governance, architecture, and engineering stakeholders.
Benefits
- Hybrid work model balancing in-person collaboration and remote working.
- Health insurance, company bonus scheme, 401(k) with company match, paid holidays, paid time off, paid volunteer days, tuition reimbursement, paid parental leave, employee shares program, and employee discounts.
- Career development, digital learning programs, and international career mobility opportunities.
Tech Stack
Apache KafkaApache SparkDatabricksDockerGitHub ActionsMicrosoft SQL ServerOracle DatabasePythonSQLTerraform
Categories
Data Engineering
About Allianz SE
Allianz SE is a Munich-based public company that provides property-casualty, life and health insurance, and asset management to individuals and businesses worldwide. It earns premiums and investment-management fees, operating through brands such as Allianz Commercial and asset managers including PIMCO and Allianz Global Investors. Founded in 1890 and listed on the Frankfurt Stock Exchange, it serves over 100 million customers in about 70 countries.