2 months ago
Bengaluru, IndiaSenior
Responsibilities
- Design, develop, maintain, and optimize high-performance ETL/ELT pipelines using PySpark, Spark SQL, Databricks, and Azure cloud components.
- Build and orchestrate data workflows in Azure and implement hybrid integrations between on-premise databases and Azure Databricks.
- Optimize Spark jobs for performance, scalability, and cost efficiency.
- Implement data quality, governance, documentation, CI/CD, automation, and version-control practices.
- Collaborate with data analysts, data scientists, and business users to define data requirements.
- Perform root cause analysis, troubleshoot issues, and ensure data pipeline reliability.
Requirements
- Bachelor’s degree in Computer Science, Engineering, or a related field.
- At least 4 years of hands-on experience in data engineering and data pipeline development in cloud environments.
- Proficiency in PySpark, Spark SQL, distributed processing, SQL, data modeling, and performance tuning.
- Strong knowledge of Azure services including ADF, Databricks, and ADLS.
- Familiarity with Git, CI/CD pipelines, and agile practices.
- Preferred qualifications include Airflow, Delta Lake, Unity Catalog, Azure Data Services, Kafka, Event Hub, HVR, APIs, cloud-native architectures, S/4 HANA, and BDC experience.
Tech Stack
Categories
Data Engineering