
Senior Azure Data Engineer
Datavail, LLC1 month ago
Mumbai, IndiaSenior
Responsibilities
- Serve as a trusted technical advisor by translating business requirements into data platform designs and recommending appropriate architecture patterns.
- Lead migrations from SSIS, Informatica, on-premises SQL, Azure Synapse, legacy ADF environments, and data warehouses to Microsoft Fabric and Databricks.
- Design, develop, and implement scalable metadata-driven data pipelines using Azure Data Factory, Microsoft Fabric Data Factory, and Azure Databricks.
- Build and maintain lakehouses, enterprise data warehouses, subject-area data marts, dimensional models, and Power BI-ready data products.
- Develop batch, incremental, streaming, hybrid-ingestion, ETL, and ELT workflows from databases, ERP and CRM systems, APIs, SaaS applications, files, and cloud storage.
- Implement security using Key Vault, managed identities, RBAC, service principals, private endpoints, VNet integration, and row- and column-level security.
- Provide production support for Azure, Fabric, and Databricks workloads, including on-call coverage, incident and change management, root-cause analysis, and SLA-driven remediation.
- Design data quality, validation, reconciliation, logging, auditing, monitoring, retry, alerting, and observability frameworks.
- Monitor pipeline performance, Fabric capacity, Databricks compute, and platform costs while recommending optimization opportunities.
- Build reusable ingestion frameworks, templates, libraries, and engineering standards across client environments.
- Conduct discovery and onboarding for new clients, assess current-state platforms, document architectures, and lead modernization roadmaps.
- Collaborate with client stakeholders, BI developers, analysts, and onshore/offshore teams to deliver reliable data and communicate technical findings.
- Mentor junior engineers and coordinate handoffs, prioritization, fixes, and enhancements.
Requirements
- 12+ years of overall experience, including 6+ years of hands-on Azure Data Factory and Databricks experience and 2+ years delivering production workloads with Microsoft Fabric.
- Bachelor’s degree.
- Expert-level PySpark and Spark SQL skills, including Spark UI and query-plan analysis and resolution of shuffle, spill, skew, and small-file issues.
- Advanced SQL and T-SQL skills covering complex joins, window functions, CTEs, MERGE logic, stored procedures, and query tuning.
- Strong Python skills for modular data-engineering code, reusable libraries, REST API ingestion, and pytest unit testing.
- Production experience building metadata- and configuration-driven ADF frameworks, watermark-based incremental loads, CDC patterns, and Self-hosted Integration Runtime configurations.
- Hands-on Delta Lake experience with MERGE/upsert, SCD Type 1 and Type 2, schema enforcement and evolution, OPTIMIZE, Z-Ordering, liquid clustering, VACUUM, and Time Travel.
- Production experience with Auto Loader, Structured Streaming, or Delta Live Tables, including checkpointing and late-arriving data.
- Hands-on Unity Catalog administration covering metastores, catalogs, schemas, grants, external locations, storage credentials, and lineage.
- Experience building Fabric Lakehouses and Warehouses, OneLake shortcuts, mirroring, notebooks, Dataflows Gen2, Direct Lake semantic models, and Fabric capacity troubleshooting.
- Experience with dimensional modeling, including star and snowflake schemas, fact and dimension design, surrogate keys, and medallion layering.
- Azure platform experience with ADLS Gen2, Azure SQL, Key Vault, managed identities, service principals, private endpoints, and VNet integration.
- CI/CD and infrastructure-as-code experience with Git, Azure DevOps YAML pipelines or GitHub Actions, ARM or Bicep templates, Databricks Asset Bundles, Fabric deployment pipelines, and Terraform or Bicep.
- Experience implementing data quality and observability using validation, reconciliation, alerting, DLT expectations or Great Expectations, Azure Monitor, and Log Analytics.
- Experience migrating SSIS, Informatica, or Synapse workloads, including code conversion, validation, and cutover.
- Hands-on production support experience with failed pipelines, Spark jobs, root-cause analysis, incidents, and changes using ITSM tools such as ServiceNow.
- Experience with Snowflake, including Snowpipe, Streams and Tasks, Time Travel, RBAC, and performance tuning.
- Experience with SSIS, SSRS, Fabric Real-Time Intelligence, Event Hubs, Kafka, Power BI, and DAX.
- Experience working in a managed services model with onshore and offshore teams and communicating technical concepts to client stakeholders.
- Preferred certifications include Microsoft Fabric Data Engineer Associate, Fabric Analytics Engineer Associate, Azure Data Engineer Associate, Databricks Certified Data Engineer Professional, or SnowPro Core.
Benefits
- Full-time position with full medical benefits.
- Hybrid work arrangement in Mumbai or Bengaluru, India.
- No travel required.
Tech Stack
Categories
Data Engineering
About Datavail, LLC
Datavail is an IT services consultancy and managed services provider that helps enterprises run and modernize data platforms, analytics, cloud, and enterprise applications. It delivers database administration and data management across Oracle, Microsoft SQL Server, MySQL, and MongoDB, plus BI/DW, analytics, and application managed services. Founded in 2007 and headquartered in Boulder, Colorado, the company is privately held and backed by private-equity owner CIVC Partners.