1 month ago
Cairo, EgyptSenior
Responsibilities
- Design, develop, test, and maintain scalable batch and streaming data pipelines using PySpark.
- Build and optimize ETL/ELT processes for large-volume enterprise data workloads.
- Develop reusable ingestion, transformation, and validation frameworks.
- Implement data quality controls, reconciliation checks, monitoring, and operational logging.
- Support dimensional, data warehouse, data lake, and lakehouse data modeling.
- Apply Spark performance optimization techniques including partitioning, join optimization, caching, and data-skew handling.
- Develop, configure, monitor, troubleshoot, and run Spark workloads within Cloudera Data Platform.
- Tune PySpark jobs, Spark configurations, SQL queries, and Hive/Impala workloads.
- Apply security, access-control, metadata, and lineage practices using Ranger and Atlas.
- Contribute to lakehouse implementations using modern table formats such as Apache Iceberg.
- Participate in requirements analysis, solution design, technical estimation, design reviews, integration testing, UAT, deployment, and operational readiness.
- Produce technical documentation, deliver knowledge-transfer sessions, and collaborate with distributed technical and business stakeholders.
- Mentor junior engineers and contribute to team capability building.
Requirements
- Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related discipline.
- At least 4 years of professional experience in data engineering.
- At least 3 years of hands-on PySpark or Apache Spark development experience.
- Strong production-grade PySpark/Spark engineering skills and experience delivering large-scale enterprise data pipelines and ETL/ELT solutions.
- Advanced SQL and query optimization skills, data warehousing knowledge, and data modeling fundamentals.
- Experience with Python, Linux, Git, and large-scale enterprise data platforms.
- Cloudera CDP experience is preferred but not mandatory; comparable Spark-based platform experience is accepted.
- Preferred experience includes Cloudera CDE/CDW, Hive, Impala, Ranger, Atlas, Apache Iceberg, Apache NiFi, Kafka, Apache Airflow, Denodo, Informatica, Oracle, Teradata, and Azure data services.
- Banking and financial services, digital banking, customer analytics, regulatory reporting, enterprise data warehousing, or data lakehouse implementation experience is highly desirable.
- Ability to work independently, communicate clearly with technical and non-technical stakeholders, learn new platforms quickly, and collaborate in distributed multicultural teams.
Tech Stack
Categories
Data Engineering
About Capgemini
Capgemini is a global IT services and consulting firm that delivers strategy, cloud, AI, software engineering, and managed services to large enterprises and public-sector clients. Founded in 1967 and headquartered in Paris, it is publicly traded on Euronext Paris and operates in 50+ countries. The group expanded its engineering capabilities by acquiring Altran in 2020, now operating as Capgemini Engineering.