26 days ago
Remote, MexicoSenior
Responsibilities
- Build and optimize Spark pipelines on Microsoft Fabric or Databricks.
- Diagnose performance using Spark UI and Delta statistics.
- Manage Delta Lake design, Fabric capacity consumption, compute sizing, throttling exposure, and operational reliability.
- Create reusable Python libraries and Spark modules for ingestion, governance, security, monitoring, and optimization.
- Troubleshoot Spark skew, spills, out-of-memory errors, task execution, and query performance.
- Optimize Delta Lake through partitioning, liquid clustering, Z-Order, OPTIMIZE, VACUUM, and MERGE strategies.
- Design, optimize, and troubleshoot large-scale data pipelines.
- Build production-grade data solutions with testing, logging, monitoring, and documentation.
Requirements
- At least 5 years of experience working as a data engineer, ETL engineer, or in similar roles.
- Strong production experience with Microsoft Fabric or Databricks, including performance tuning and operational support.
- Experience troubleshooting Apache Spark performance issues, including skew, spills, out-of-memory errors, task execution, and query optimization.
- Hands-on experience with Delta Lake optimization techniques, including partitioning, liquid clustering, Z-Order, OPTIMIZE, VACUUM, and MERGE.
- Strong SQL and data engineering skills for large-scale pipeline design and troubleshooting.
- Python development experience building reliable, production-grade data solutions.
- Fluent English proficiency.
Benefits
- Work on meaningful technology projects addressing real-world challenges.
- Join a company focused on sustainability, inclusion, and digital equity.
- Contribute to initiatives supporting people, planet, and society.
- Work remotely from any state in Mexico.
Tech Stack
Categories
Data Engineering
About Capgemini
Capgemini is a global IT services and consulting firm that delivers strategy, cloud, AI, software engineering, and managed services to large enterprises and public-sector clients. Founded in 1967 and headquartered in Paris, it is publicly traded on Euronext Paris and operates in 50+ countries. The group expanded its engineering capabilities by acquiring Altran in 2020, now operating as Capgemini Engineering.