7 days ago
Vila Nova de Gaia, PortugalMid Level
Responsibilities
- Build and maintain batch, CDC, and streaming data pipelines across SAP, Kafka, APIs, and databases.
- Develop scalable data solutions with Azure Data Factory, Databricks, Spark, SQL, and Delta Lake.
- Implement data-quality controls, monitoring, observability, alerting, and data contracts.
- Optimize pipeline performance, cost efficiency, and security using Unity Catalog.
- Contribute reusable frameworks, CI/CD processes, data engineering practices, code reviews, and continuous improvement.
- Collaborate with DataOps, ML Engineering, AI, Analytics Engineering, and cross-functional teams to deliver curated data for analytics and AI products.
Requirements
- Proficiency in Python or Scala and SQL for production data pipelines.
- Practical experience with Spark, Delta Lake, and Databricks, including performance optimization.
- Experience with batch pipelines and at least one change-data-capture or streaming pattern.
- Practical experience with Azure data services, including Azure Data Factory.
- Experience with automated data-quality testing and pipeline monitoring.
- Working knowledge of Git, CI/CD, code review, Unity Catalog, or comparable data-governance and access-control tooling.
- Experience with Confluent Kafka, Spark Structured Streaming, or comparable streaming frameworks is preferred.
- Knowledge of SAP S/4HANA, BW, Business Data Cloud extraction and data structures, data contracts, high-volume transactional data, data mesh principles, document processing, embeddings, and vector indexes is preferred.
Benefits
- Flexible hybrid work arrangement with the ability to manage working hours and work from the office or remotely according to company policy.
- Health and life insurance, referral program bonuses, and other partnership-related benefits.
- Career Acceleration Programs supporting professional growth, reskilling, and new skills development.
- Collaborative, empowering work environment with autonomy and peer relationships.
- Role contributes to the launch of Capgemini’s new Tech Hub in Porto.
Tech Stack
Categories
Data Engineering
About Capgemini
Capgemini is a global IT services and consulting firm that delivers strategy, cloud, AI, software engineering, and managed services to large enterprises and public-sector clients. Founded in 1967 and headquartered in Paris, it is publicly traded on Euronext Paris and operates in 50+ countries. The group expanded its engineering capabilities by acquiring Altran in 2020, now operating as Capgemini Engineering.