Senior Data Engineer
NorthBay Solutions3 months ago
Remote, IndiaSenior
Responsibilities
- Design, develop, and maintain scalable data platforms and data pipelines on Microsoft Azure.
- Build and optimize batch and real-time data ingestion frameworks for structured and unstructured sources.
- Design and implement data lake, data warehouse, and lakehouse architectures for analytics and reporting.
- Develop and manage ETL/ELT processes and data integration solutions.
- Implement data transformation, cleansing, validation, quality, governance, security, monitoring, and compliance frameworks.
- Collaborate with business stakeholders, data analysts, data scientists, and application teams to deliver scalable data solutions.
- Optimize data storage, processing, and query performance across enterprise data platforms.
- Support integration with AI/ML systems, business intelligence tools, and enterprise applications.
- Participate in architecture reviews, code reviews, troubleshooting, and technical mentoring.
- Ensure high availability, scalability, and operational excellence of data platforms and pipelines.
Requirements
- 8–10+ years of experience in data engineering, data warehousing, and enterprise data platform development.
- Strong hands-on experience with Microsoft Azure Data Services, Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage Gen2, and Azure SQL Database.
- Experience building and managing large-scale ETL/ELT pipelines and data integration solutions.
- Strong SQL proficiency, including query optimization and database performance tuning.
- Hands-on experience with PySpark, Apache Spark, and distributed data processing frameworks.
- Strong programming skills in Python, Scala, or Java.
- Experience with dimensional modeling, data warehousing, lakehouse architectures, and structured, semi-structured, and unstructured data.
- Understanding of data governance, data quality, metadata management, and security best practices.
- Experience with REST APIs, data integration patterns, and enterprise system connectivity.
- Hands-on experience with Git, CI/CD pipelines, and DevOps practices.
- Preferred experience with Microsoft Fabric, OneLake, Dataflows, Fabric Data Engineering, Databricks, Delta Lake, Azure Event Hubs, Apache Kafka, Azure Stream Analytics, Power BI datasets, semantic models, Microsoft Purview, and AI/ML workloads.
- Microsoft Azure Data Engineering certifications are highly preferred.
- Nice-to-have experience includes MLOps, DataOps, Docker, Kubernetes, Terraform, ARM Templates, Bicep, Snowflake, BigQuery, enterprise data migration, modernization projects, Generative AI, and Vector Databases.
- Strong analytical, problem-solving, and communication skills, with experience working in Agile/Scrum environments.
Benefits
- Remote work from India.
- Full-time, permanent employment.
Tech Stack
Apache KafkaApache SparkAzureDatabricksDockerGitGoogle BigQueryJavaKubernetesPythonScalaSnowflakeSQLTerraform
Categories
Data Engineering
About NorthBay Solutions
NorthBay Solutions is a privately held IT services firm that helps enterprises modernize on AWS through cloud migration, data engineering, and AI/ML solutions. An AWS Premier Consulting Partner founded in 2010 and headquartered in Andover, MA, it also partners with Snowflake and provides managed services, DevOps enablement, and application development. The company operates distributed teams across the US, Canada, UAE, India, and Pakistan.