26 days ago
Remote, MexicoSenior
Responsibilities
- Build and optimize Spark pipelines on Microsoft Fabric or Databricks.
- Diagnose performance using Spark UI and Delta statistics.
- Manage Delta Lake design, Fabric capacity consumption, compute sizing, throttling exposure, and operational reliability.
- Create reusable Python libraries and Spark modules for ingestion, governance, security, monitoring, and optimization.
- Optimize Delta Lake using partitioning, liquid clustering, Z-Order, OPTIMIZE, VACUUM, and MERGE strategies.
- Design, optimize, and troubleshoot large-scale data pipelines.
- Develop reliable production-grade data solutions with testing, logging, monitoring, and documentation.
Requirements
- At least 5 years of experience as a data engineer, ETL engineer, or in a similar role.
- Strong production experience with Microsoft Fabric or Databricks, including performance tuning and operational support.
- Experience troubleshooting Apache Spark issues such as skew, spills, out-of-memory errors, task execution problems, and query optimization.
- Hands-on experience optimizing Delta Lake with partitioning, liquid clustering, Z-Order, OPTIMIZE, VACUUM, and MERGE.
- Strong SQL and data engineering skills for large-scale data pipelines.
- Python development experience and the ability to build reliable production-grade data solutions.
- Fluent English level.
Benefits
- Work on meaningful technology projects that solve real-world challenges.
- Join a company focused on sustainability, inclusion, and digital equity.
- Contribute to initiatives aimed at improving outcomes for people, planet, and society.
- This role can be performed from any state in Mexico.
Tech Stack
Categories
Data Engineering
About Capgemini
Capgemini is a global IT services and consulting firm that delivers strategy, cloud, AI, software engineering, and managed services to large enterprises and public-sector clients. Founded in 1967 and headquartered in Paris, it is publicly traded on Euronext Paris and operates in 50+ countries. The group expanded its engineering capabilities by acquiring Altran in 2020, now operating as Capgemini Engineering.