Capgemini

Senior Data Engineer (Python/Spark/ MS Fabric)

Capgemini
Apply
2 days ago
Remote, Ukraine +4 moreSenior

Responsibilities

  • Design, optimize, and operate reliable, large-scale, cost-aware Spark and Delta Lake workloads.
  • Build and optimize Spark pipelines on Microsoft Fabric or Databricks.
  • Diagnose production performance using Spark UI and Delta statistics.
  • Manage Delta Lake design, Fabric capacity consumption, compute sizing, throttling exposure, and operational reliability.
  • Create reusable Python libraries and Spark modules for ingestion, governance, security, monitoring, and optimization.
  • Measure workload behavior before changing infrastructure and optimize code and configuration before increasing compute.
  • Document pool sizing, throttling risk, operational requirements, and capacity behavior for platform and engineering teams.
  • Design operational pipelines with idempotency, watermarks, quarantine, monitoring, failure recovery, and SLA awareness.

Requirements

  • Production engineering experience with Microsoft Fabric or Databricks.
  • Hands-on Apache Spark and Spark UI troubleshooting experience, including shuffle skew, spill, OOM, task-duration analysis, AQE, join strategy, and predicate pushdown.
  • Experience optimizing Delta Lake through liquid clustering, partitioning, Z-order, file sizing, OPTIMIZE, statistics, VACUUM, and incremental MERGE.
  • Experience with Fabric capacity economics, including CU-seconds, bursting, smoothing, throttling stages, pool configuration, node sizing, concurrency, and workload isolation.
  • Production Python library development experience covering packaging, validation, logging, testing, versioning, and documentation.
  • Experience designing operationally reliable data pipelines.
  • Preferred experience with OneLake security and access roles, governance metadata and TBLPROPERTIES, reusable ingestion and maintenance frameworks, secrets handling, thread safety, and Direct Lake optimization.

Benefits

  • Health insurance from the first days of employment, regardless of the probationary period.
  • Company Christmas holidays from December 25 through December 31.
  • Support for psychological counseling through Veteran HUB and cooperation with the Superhumans center.
  • Must be available to overlap with the customer's US Eastern Time working hours through approximately 4:30 PM ET.

Tech Stack

Categories

Data Engineering
Capgemini

About Capgemini

10,000+ employees

Capgemini is a global IT services and consulting firm that delivers strategy, cloud, AI, software engineering, and managed services to large enterprises and public-sector clients. Founded in 1967 and headquartered in Paris, it is publicly traded on Euronext Paris and operates in 50+ countries. The group expanded its engineering capabilities by acquiring Altran in 2020, now operating as Capgemini Engineering.

Contact me