2 days ago
Remote, Ukraine +4 moreSenior
Responsibilities
- Design, optimize, and operate reliable, large-scale, cost-aware Spark and Delta Lake workloads.
- Build and optimize Spark pipelines on Microsoft Fabric or Databricks.
- Diagnose production performance using Spark UI and Delta statistics.
- Manage Delta Lake design, Fabric capacity consumption, compute sizing, throttling exposure, and operational reliability.
- Create reusable Python libraries and Spark modules for ingestion, governance, security, monitoring, and optimization.
- Measure workload behavior before changing infrastructure and optimize code and configuration before increasing compute.
- Document pool sizing, throttling risk, operational requirements, and capacity behavior for platform and engineering teams.
- Design operational pipelines with idempotency, watermarks, quarantine, monitoring, failure recovery, and SLA awareness.
Requirements
- Production engineering experience with Microsoft Fabric or Databricks.
- Hands-on Apache Spark and Spark UI troubleshooting experience, including shuffle skew, spill, OOM, task-duration analysis, AQE, join strategy, and predicate pushdown.
- Experience optimizing Delta Lake through liquid clustering, partitioning, Z-order, file sizing, OPTIMIZE, statistics, VACUUM, and incremental MERGE.
- Experience with Fabric capacity economics, including CU-seconds, bursting, smoothing, throttling stages, pool configuration, node sizing, concurrency, and workload isolation.
- Production Python library development experience covering packaging, validation, logging, testing, versioning, and documentation.
- Experience designing operationally reliable data pipelines.
- Preferred experience with OneLake security and access roles, governance metadata and TBLPROPERTIES, reusable ingestion and maintenance frameworks, secrets handling, thread safety, and Direct Lake optimization.
Benefits
- Health insurance from the first days of employment, regardless of the probationary period.
- Company Christmas holidays from December 25 through December 31.
- Support for psychological counseling through Veteran HUB and cooperation with the Superhumans center.
- Must be available to overlap with the customer's US Eastern Time working hours through approximately 4:30 PM ET.
Tech Stack
Categories
Data Engineering
About Capgemini
Capgemini is a global IT services and consulting firm that delivers strategy, cloud, AI, software engineering, and managed services to large enterprises and public-sector clients. Founded in 1967 and headquartered in Paris, it is publicly traded on Euronext Paris and operates in 50+ countries. The group expanded its engineering capabilities by acquiring Altran in 2020, now operating as Capgemini Engineering.
