Data Engineer - AI Pipelines & DataOps
Intersnack IT KG7 months ago
Düsseldorf, GermanySenior
Responsibilities
- Design and implement scalable ingestion pipelines for structured and unstructured data from manufacturing systems, edge devices, and enterprise platforms.
- Build and maintain batch and streaming pipelines for analytics and AI use cases using cloud-native tooling on Microsoft Azure and/or AWS.
- Design and expose REST or GraphQL APIs for data assets, machine learning model endpoints, and AI services.
- Implement DataOps and CI/CD practices, including automated testing, versioning, and release management for data infrastructure.
- Ensure data quality, lineage, observability, monitoring, and alerting across pipelines.
- Integrate IoT and operational technology data into the central data platform while addressing latency, format, and volume challenges.
- Collaborate with data scientists and AI engineers on data flows for model training, inference, and knowledge retrieval pipelines.
- Apply security-by-design practices involving access control, encryption, and data-leakage protection.
- Support colleagues through AI literacy and enablement activities related to pipeline health and data quality.
Requirements
- Proven experience designing and implementing production-grade data pipelines using Microsoft Azure and/or AWS cloud-native services.
- Strong proficiency in a pipeline or transformation framework such as Apache Spark, dbt, Apache Kafka, or equivalent, plus Python or Scala.
- Hands-on experience with SQL and NoSQL databases, including data modelling and query optimisation for transactional and analytical workloads.
- Experience building and maintaining streaming and batch pipelines for analytics and AI applications.
- Ability to design and implement REST or GraphQL APIs with versioning, documentation, and governance.
- Familiarity with DataOps, CI/CD, automated testing, pipeline orchestration, and infrastructure-as-code.
- Experience with data quality tooling, lineage tracking, and observability frameworks such as Great Expectations or OpenLineage.
- Working knowledge of AI and data security fundamentals, including access controls, encryption, and data-exfiltration risks.
- Awareness of GDPR, the EU AI Act, and EU data sovereignty requirements.
- Strong spoken and written English; German is an advantage.
- Experience with manufacturing or operational technology data, including IoT or edge-device ingestion and time-series data, is valuable.
- Familiarity with Microsoft Fabric, OneLake, or Azure Purview is valuable.
- Exposure to MLOps, data versioning, feature stores, or model monitoring pipelines is valuable.
- Experience with Terraform or other infrastructure-as-code tooling is valuable.
- FMCG, manufacturing, or supply-chain experience is valuable.
Benefits
- Düsseldorf is the home base with flexibility for remote working.
- The role offers autonomy to define pipeline patterns, API standards, and DataOps practices across a €4.5 billion business.
- The position provides exposure to manufacturing edge devices, IoT sensors, enterprise data systems, AI, analytics, and international business operations.
- A valid work and residence permit is required for non-EU applicants.
Tech Stack
Categories
Data Engineering