Senior Data Engineer
Baker Tilly3 months ago
Bengaluru, IndiaSenior / Staff+
Responsibilities
- Design, build, and maintain scalable ETL/ELT pipelines on Azure Databricks across bronze, silver, gold, and platinum medallion layers.
- Ingest structured, semi-structured, and unstructured data using Spark Structured Streaming, Auto Loader, Lakeflow Connect, JDBC/CDC, Databricks Marketplace, and Delta Sharing.
- Build and manage Unity Catalog assets, Volumes, external locations, and governed access to ADLS Gen2.
- Create audit, control, operational-metadata, lineage, reconciliation, and data-quality mechanisms for pipeline observability.
- Define and apply data governance strategies covering access control, classification, PII handling, lineage, retention, and data quality.
- Develop performant Delta Lake data models using partitioning, Z-ordering or liquid clustering, and OPTIMIZE.
- Prepare analytics- and AI-ready datasets supporting feature engineering, feature stores, embeddings, vector data, and ML-ready data products.
- Enable cross-cloud data movement and sharing from AWS, GCP, and cloud data warehouses into the Azure Databricks lakehouse.
- Troubleshoot pipeline and data-quality issues and partner with BI, analytics, and business teams on Power BI and Tableau datasets.
- Orchestrate workflows with Databricks Workflows/Jobs and Lakeflow Declarative Pipelines, and contribute to Git- and Databricks Asset Bundles-based CI/CD.
- Contribute to engineering standards, documentation, code reviews, testing, project delivery, status reporting, risk management, and continuous improvement.
Requirements
- Bachelor’s degree required, preferably in Information Technology, Computer Science, Data Science, Analytics, or Statistics.
- 5–8 years of hands-on experience in data engineering, data integration, or analytics engineering, including substantial experience delivering on Databricks.
- Deep hands-on experience with Azure Databricks workspaces, clusters or SQL warehouses, Unity Catalog, and ADLS Gen2 integration.
- Strong proficiency in Spark/PySpark and SQL, including complex queries and advanced data transformations.
- Experience building medallion-architecture pipelines on Delta Lake.
- Experience ingesting data through Structured Streaming, Auto Loader, connectors, Databricks Marketplace, and Delta Sharing.
- Experience with Unity Catalog governance, access control, lineage, data classification, Volumes, and external locations.
- Experience designing audit/control tables, operational-metadata tables, observability mechanisms, reconciliation processes, and data-quality frameworks.
- Experience migrating or sharing data across AWS, GCP, Snowflake, Redshift, BigQuery, and Databricks lakehouses.
- Familiarity with Databricks Workflows/Jobs, DLT, Git, and Databricks Asset Bundles in data engineering environments.
- Understanding of data governance, data-quality frameworks, and metric standardization, with experience supporting AI, advanced analytics, or automation initiatives.
- Strong analytical, problem-solving, independent project delivery, communication, and risk-management skills.
- Preferred experience handling tax, assurance, and financial data and related confidentiality and regulatory considerations.
- Preferred exposure to MLflow, feature stores, Mosaic AI, Vector Search, Power BI, Tableau, and semantic-layer design.
- Databricks certification, such as Databricks Certified Data Engineer Professional, is preferred.
Tech Stack
Categories
Data Engineering
About Baker Tilly
Baker Tilly provides audit/assurance, tax, and advisory services—including analytics, governance risk and compliance, and government contracting—to companies, public sector entities, and nonprofits. Headquartered in Chicago and organized as a partnership, it operates coast to coast in the U.S. and is an independent member of Baker Tilly International, with U.S. attest services delivered by Baker Tilly US, LLP and tax and advisory by Baker Tilly Advisory Group, LP.