3 days ago
Opole, Poland +6 moreSenior
Responsibilities
- Design, optimize, and operate reliable, large-scale, cost-aware Spark and Delta Lake workloads.
- Build and optimize Spark pipelines on Microsoft Fabric or Databricks and diagnose performance using Spark UI and Delta statistics.
- Manage Delta Lake design, Fabric capacity consumption, compute sizing, throttling exposure, and operational reliability.
- Create reusable Python libraries and Spark modules for ingestion, governance, security, monitoring, and optimization.
- Analyze workload behavior and optimize code and configuration before increasing infrastructure capacity.
- Document pool sizing, throttling risk, operational requirements, and capacity behavior for platform and engineering teams.
- Design operational pipelines with idempotency, watermarks, quarantine, monitoring, failure recovery, and SLA awareness.
Requirements
- Production engineering experience with Microsoft Fabric or Databricks.
- Strong Apache Spark and Spark UI troubleshooting experience, including shuffle skew, spill, out-of-memory issues, task-duration analysis, AQE, join strategy, and predicate pushdown.
- Experience optimizing Delta Lake with liquid clustering, partitioning, Z-order, file sizing, OPTIMIZE, statistics, VACUUM, and incremental MERGE.
- Experience with Fabric capacity economics, including CU-seconds, bursting, smoothing, throttling stages, pool configuration, node sizing, concurrency, and workload isolation.
- Production Python library development experience covering packaging, validation, logging, testing, versioning, and documentation.
- Demonstrated production troubleshooting and capacity-economics experience beyond notebook or ETL development.
- Availability to overlap with US Eastern Time working hours through approximately 4:30 PM ET.
Benefits
- Medical care with Medicover, private life insurance, sports card, and Capgemini Helpline therapeutic support.
- Access to more than 70 training tracks and certification opportunities through the NEXT platform, plus Education First, TED Talks, and Udemy Business resources.
- Continuous feedback and performance discussions supported by the GetSuccess performance management tool.
- Hybrid working model after onboarding, with a modern office and home-office package including a laptop, monitor, and chair.
Tech Stack
Categories
Data Engineering
About Capgemini
Capgemini is a global IT services and consulting firm that delivers strategy, cloud, AI, software engineering, and managed services to large enterprises and public-sector clients. Founded in 1967 and headquartered in Paris, it is publicly traded on Euronext Paris and operates in 50+ countries. The group expanded its engineering capabilities by acquiring Altran in 2020, now operating as Capgemini Engineering.
