Senior Data Engineer
NorthBay Solutions24 days ago
Bengaluru, IndiaSenior
Responsibilities
- Design, develop, and maintain scalable Microsoft Azure data platforms and data pipelines.
- Build and optimize batch and real-time ingestion frameworks for structured, semi-structured, and unstructured data.
- Design and implement data lake, data warehouse, and lakehouse architectures for analytics and reporting.
- Develop, manage, and optimize ETL/ELT pipelines and data integration frameworks.
- Implement data transformation, cleansing, validation, quality, governance, security, monitoring, and metadata management practices.
- Collaborate with business stakeholders, analysts, data scientists, and application teams to translate requirements into scalable data solutions.
- Optimize enterprise data storage, processing, and query performance.
- Integrate data platforms with AI/ML solutions, business intelligence tools, and enterprise applications.
- Develop data models, reusable components, and efficient data processing workflows.
- Participate in architecture reviews, code reviews, troubleshooting, documentation, knowledge sharing, and mentoring.
- Support data platform reliability, scalability, high availability, and operational excellence.
- Implement version control, CI/CD pipelines, and DevOps practices for data engineering workloads.
Requirements
- Bachelor’s degree in Computer Science, Information Technology, Software Engineering, or a related field.
- 6–10 years of professional experience in Data Engineering, Data Warehousing, or Enterprise Data Platform development.
- Strong hands-on experience with Microsoft Azure Data Services and cloud-based data engineering solutions.
- Practical experience with Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage Gen2, and Azure SQL Database.
- Experience designing, developing, and maintaining large-scale ETL/ELT pipelines and data integration frameworks.
- Strong SQL skills, including query optimization and database performance tuning.
- Hands-on experience with PySpark, Apache Spark, and distributed data processing frameworks.
- Strong Python programming skills; Scala or Java exposure is advantageous.
- Understanding of dimensional modeling, data warehousing, and modern lakehouse architectures.
- Experience processing structured, semi-structured, and unstructured data.
- Knowledge of data governance, data quality, metadata management, and data security practices.
- Experience with REST APIs, enterprise data integration patterns, and business application connectivity.
- Familiarity with Git, CI/CD pipelines, and DevOps practices.
- Strong analytical, troubleshooting, problem-solving, communication, and collaboration skills.
- Preferred experience with Microsoft Fabric, OneLake, Dataflows, Databricks, Delta Lake, and lakehouse implementations.
- Preferred knowledge of Azure Event Hubs, Apache Kafka, Azure Stream Analytics, Power BI datasets, semantic models, Microsoft Purview, and enterprise reporting architectures.
- Experience supporting AI/ML and advanced analytics workloads is preferred.
- Microsoft Azure Data Engineering certifications are highly desirable.
- Nice-to-have experience includes DataOps, MLOps, Docker, Kubernetes, Terraform, ARM templates, Bicep, Snowflake, Google BigQuery, enterprise data migration, cloud transformation, generative AI, vector databases, and AI-supporting data platforms.
Benefits
- Full-time, permanent employment.
- Onsite work in Abu Dhabi, United Arab Emirates.
- Opportunity to work on modern Azure data platforms, cloud analytics, lakehouse technologies, and enterprise data modernization.
- Exposure to business intelligence, analytics, and AI/ML data initiatives.
- Collaboration with multidisciplinary technical and business teams.
Tech Stack
Apache KafkaApache SparkAzureDatabricksDockerGitGoogle BigQueryJavaKubernetesPythonScalaSnowflakeSQLTerraform
Categories
Data Engineering
About NorthBay Solutions
NorthBay Solutions is a privately held IT services firm that helps enterprises modernize on AWS through cloud migration, data engineering, and AI/ML solutions. An AWS Premier Consulting Partner founded in 2010 and headquartered in Andover, MA, it also partners with Snowflake and provides managed services, DevOps enablement, and application development. The company operates distributed teams across the US, Canada, UAE, India, and Pakistan.