Data Engineer for AI (f/m/d)
Allianz SE28 days ago
Frankfurt, GermanyMid Level
Responsibilities
- Develop and operate reusable ingestion, transformation, validation, enrichment, and data preparation pipelines for structured and unstructured data.
- Build reliable batch and streaming pipelines on Databricks and publish trusted Bronze, Silver, and Gold data products for AI, ML, analytical, and BI consumption.
- Design and operate feature engineering pipelines and a versioned feature store with ownership, documentation, and SLA information.
- Build and maintain GenAI/RAG data pipelines, including ingestion, chunking, embedding refreshes, and index updates.
- Enable offline evaluations, A/B tests, feedback-driven retraining, and systematic model improvement loops.
- Operationalize ingestion, transformation, serving, orchestration, monitoring, and governance capabilities across the Data and AI Platform.
- Roll out Databricks Lakeflow connectors and integrate new internal, external, enterprise, SaaS, and cloud data sources.
- Implement Unity Catalog access control, lineage, retention, encryption, auditing, data quality validation, monitoring, and compliance controls.
- Define freshness and reliability SLAs/SLOs, optimize Spark workloads and storage layouts, and manage compute cost and performance.
- Collaborate with Data Scientists, ML Engineers, AI CoE, DevOps, SecOps, and platform architects to define requirements and promote reusable patterns.
- Provide L3 incident support, root-cause analysis, technical guidance, coaching, documentation, and continuous platform improvements.
- Expose curated data products securely to feature stores, model training and inference systems, and BI consumers.
Requirements
- At least 3 years of practical experience building and operating Databricks data pipelines and data products at scale, using batch and/or streaming approaches.
- Strong understanding of data engineering for AI/ML, including data modeling, data quality, data contracts, feature engineering collaboration, reproducibility, and model-data interactions.
- Expert-level skills in Databricks, Spark, Python, and SQL, with solid CI/CD and infrastructure-as-code practices such as Terraform on Azure.
- Diploma in computer science or a similar field.
- Strong understanding of data architecture and platform patterns, including Lakehouse, Delta, medallion, and governed data product approaches.
- Excellent analytical, planning, organizational, communication, documentation, and knowledge-sharing skills.
- Experience in financial services, preferably asset management, is preferred.
- International work experience is preferred.
- Databricks Certified Data Engineer Professional or equivalent certification is preferred.
- Databricks Certified Data Engineer Associate or equivalent certification is preferred.
- Databricks Certified Generative AI Engineer Associate certification is helpful for GenAI data preparation.
- Hands-on Microsoft Foundry/Azure AI Foundry experience, or willingness to build expertise, is preferred.
Benefits
- Flexible work arrangements, including a hybrid model and flexible working hours.
- Company pension and savings plans.
- Relocation and childcare support or facilities.
- Company share purchasing plan.
- Mental health and wellbeing programs.
- Mobility benefits including Jobrad bike leasing and subsidized Jobticket.
- Career opportunities across the Allianz Group.
- Self-guided learning and development opportunities.
- Volunteering time.
- The position is based in Frankfurt and is a full-time, permanent role.
Tech Stack
Categories
Data Engineering
About Allianz SE
Allianz SE is a Munich-based public company that provides property-casualty, life and health insurance, and asset management to individuals and businesses worldwide. It earns premiums and investment-management fees, operating through brands such as Allianz Commercial and asset managers including PIMCO and Allianz Global Investors. Founded in 1890 and listed on the Frankfurt Stock Exchange, it serves over 100 million customers in about 70 countries.