7 hours ago
Mumbai, IndiaSenior
Responsibilities
- Own the end-to-end architecture, strategy, and operational maturity of the Databricks Lakehouse platform.
- Design workspace topology, cluster policies, compute strategy, multi-cloud deployment patterns, and platform reliability practices.
- Define and enforce Unity Catalog governance, including catalogs, schemas, access controls, lineage, audit logging, and metadata standards.
- Establish data engineering standards for Delta Lake, Delta Live Tables, medallion architecture, Z-ordering, liquid clustering, partitioning, and file compaction.
- Industrialize platform provisioning and pipeline deployment through Terraform or Databricks Asset Bundles, CI/CD tooling, and monitoring.
- Own Databricks cost transparency and optimization through right-sizing, workload isolation, chargeback/showback, and budget alerting.
- Architect AI-native capabilities such as feature stores, vector search, and GenAI-ready pipelines within governed platform boundaries.
- Govern the rollout of Databricks Genie, AI/BI Dashboards, Mosaic AI, and other native AI use cases.
- Partner with engineering, analytics, data science, CloudOps, DevOps, and business stakeholders to deliver self-service platform capabilities.
- Lead a Databricks community of practice through standards, enablement sessions, mentorship, and delivery support.
Requirements
- 8–10+ years of experience in data engineering, data platform architecture, or distributed systems, including 3+ years architecting and operating Databricks environments at scale.
- Direct ownership and hands-on experience building, managing, running, and scaling a production-grade, large-scale, multi-tenant Databricks Data Platform.
- Expertise with Unity Catalog, Delta Lake, Delta Live Tables, Databricks Workflows, Databricks SQL, and cluster and compute policy design.
- Experience architecting AI-native platform capabilities, including feature stores, vector search or indexes, and GenAI-ready pipelines.
- Experience using AI-assisted and agentic engineering practices, such as coding copilots and automated testing or operations agents.
- Deep hands-on expertise with Apache Spark batch and structured streaming for large-scale processing and performance tuning.
- Expert-level proficiency in Python and SQL, with working knowledge of dbt.
- Experience with Terraform or Databricks Asset Bundles and CI/CD tooling for infrastructure and pipeline automation.
- Strong understanding of data security, RBAC, ABAC, PII handling, and compliance requirements in governed lakehouse environments.
- Ability to evaluate architectural trade-offs, identify technical or cost risks, and communicate decisions clearly to technical and non-technical stakeholders.
- Preferred qualifications include Databricks Certified Data Engineer Professional or Databricks Certified Platform Architect certification.
- Ability to use a Bottom Line Up Front communication style and explain the rationale behind platform decisions.
Tech Stack
Categories
Data EngineeringDevOps
About Nielsen
Nielsen provides audience measurement and analytics that quantify who watches, listens, and buys across TV, digital, radio, and streaming. It sells ratings, cross-platform datasets, and advertising effectiveness tools to media owners, advertisers, and agencies on subscription and consulting agreements. Founded in 1923 and headquartered in New York, it operates in 55+ countries and is known for U.S. TV ratings.