Senior Lead Data Engineer - Databricks
Integrant, Inc.4 months ago
Cairo, EgyptStaff+
Responsibilities
- Lead the design, development, maintenance, and optimization of scalable data engineering solutions using Azure and Databricks.
- Design and implement Lakehouse architectures, Data Warehouses, Data Lakes, and Delta Lake solutions.
- Build and optimize large-scale batch and streaming data pipelines using Databricks, PySpark, SQL, and Azure Data Factory.
- Optimize Spark workloads, including partitioning, joins, caching, performance, scalability, and cost efficiency.
- Implement data governance, security, metadata management, and enterprise data platform practices using Azure services.
- Collaborate with clients to gather requirements and translate business needs into technical solutions.
- Establish CI/CD pipelines and DevOps practices for data engineering workloads.
- Mentor and coach junior data engineers and promote engineering best practices.
- Participate in architecture discussions, solution design, technical leadership, knowledge sharing, and continuous improvement.
Requirements
- 10+ years of professional data engineering experience.
- 4+ years of hands-on Databricks experience.
- Strong experience with PySpark, Apache Spark, SQL, Python, Lakehouse architectures, Delta Lake, and Medallion Architecture.
- Experience with Azure Data Factory, Azure Data Lake Storage Gen2, Azure Synapse Analytics, data warehousing, dimensional modeling, ETL/ELT design patterns, and Spark optimization.
- Experience with Git, Azure DevOps CI/CD pipelines, Azure Key Vault, Microsoft Entra ID, Microsoft Purview, and scalable enterprise data platforms in Azure.
- Experience working directly with clients, gathering technical requirements, mentoring engineers, and leading technical initiatives.
- Preferred experience with Snowflake, Structured Streaming, Kafka, Event Hubs, Databricks Workflows, Delta Live Tables, Unity Catalog, Databricks SQL, MLflow, dbt, Airflow, or other orchestration platforms.
- Nice-to-have experience with DataOps, MLOps, Azure Machine Learning, Azure Cognitive Services, Docker, Kubernetes, Microsoft Fabric, and Power BI.
- An MSc in Computer Science or a related field is nice to have.
Benefits
- Salary is paid in USD.
- Hybrid workplace in Maadi, Egypt.
- Six-month career-advancing opportunities.
- Supportive and friendly work environment.
- Premium medical insurance for employees and families.
- English language development courses and technical development courses.
- Interest-free loans paid over 2.5 years.
- Planned overtime program, employment referral program, premium Maadi location, and social insurance.
Tech Stack
Categories
Data Engineering
About Integrant, Inc.
Integrant, Inc. is a privately held software engineering services firm that builds custom enterprise systems and regulated‑industry applications for global clients. Founded in 1992 and headquartered in San Diego, it delivers development, QA automation, and legacy modernization, and integrates AI agents/LLMs across its agentic SDLC. The company serves medical devices, life sciences/pharma (GxP/21 CFR Part 11), financial services, manufacturing, and public‑sector customers, with engineering centers in Cairo and Amman.