Capgemini

Senior Data Engineer

Capgemini
Apply
3 days ago
Opole, Poland +6 moreSenior

Responsibilities

  • Design, optimize, and operate reliable, large-scale, cost-aware Spark and Delta Lake workloads.
  • Build and optimize Spark pipelines on Microsoft Fabric or Databricks and diagnose performance using Spark UI and Delta statistics.
  • Manage Delta Lake design, Fabric capacity consumption, compute sizing, throttling exposure, and operational reliability.
  • Create reusable Python libraries and Spark modules for ingestion, governance, security, monitoring, and optimization.
  • Analyze workload behavior and optimize code and configuration before increasing infrastructure capacity.
  • Document pool sizing, throttling risk, operational requirements, and capacity behavior for platform and engineering teams.
  • Design operational pipelines with idempotency, watermarks, quarantine, monitoring, failure recovery, and SLA awareness.

Requirements

  • Production engineering experience with Microsoft Fabric or Databricks.
  • Strong Apache Spark and Spark UI troubleshooting experience, including shuffle skew, spill, out-of-memory issues, task-duration analysis, AQE, join strategy, and predicate pushdown.
  • Experience optimizing Delta Lake with liquid clustering, partitioning, Z-order, file sizing, OPTIMIZE, statistics, VACUUM, and incremental MERGE.
  • Experience with Fabric capacity economics, including CU-seconds, bursting, smoothing, throttling stages, pool configuration, node sizing, concurrency, and workload isolation.
  • Production Python library development experience covering packaging, validation, logging, testing, versioning, and documentation.
  • Demonstrated production troubleshooting and capacity-economics experience beyond notebook or ETL development.
  • Availability to overlap with US Eastern Time working hours through approximately 4:30 PM ET.

Benefits

  • Medical care with Medicover, private life insurance, sports card, and Capgemini Helpline therapeutic support.
  • Access to more than 70 training tracks and certification opportunities through the NEXT platform, plus Education First, TED Talks, and Udemy Business resources.
  • Continuous feedback and performance discussions supported by the GetSuccess performance management tool.
  • Hybrid working model after onboarding, with a modern office and home-office package including a laptop, monitor, and chair.

Tech Stack

Categories

Data Engineering
Capgemini

About Capgemini

10,000+ employees

Capgemini is a global IT services and consulting firm that delivers strategy, cloud, AI, software engineering, and managed services to large enterprises and public-sector clients. Founded in 1967 and headquartered in Paris, it is publicly traded on Euronext Paris and operates in 50+ countries. The group expanded its engineering capabilities by acquiring Altran in 2020, now operating as Capgemini Engineering.

Contact me