Director, Distinguished Engineer - Data Platforms
S&P Global25 days ago
Hyderābād, IndiaStaff+
Responsibilities
- Lead migration of existing data pipelines to the target Databricks enterprise data platform and establish reusable patterns for batch and streaming ingestion, transformation, testing, deployment, monitoring, and governance.
- Guide AWS cloud-native data platform architecture using services including Amazon S3, AWS Glue, AWS Lambda, Amazon Kinesis, and AWS Lake Formation.
- Influence architecture involving Databricks, Delta Lake, Apache Iceberg, Unity Catalog, metadata-driven processing, governed data lakes, semantic models, data catalogs, and lineage.
- Provide technical direction to distributed and offshore engineering teams without direct people-management responsibility.
- Support integration with enterprise data mastering platforms, including data acquisition, cleansing, enrichment, reconciliation, golden-copy generation, and trusted downstream distribution.
- Develop semantic modeling practices covering business definitions, entities, attributes, relationships, metrics, business terms, and governed data consumption.
- Champion AI-assisted engineering with Claude, GitHub Copilot, LLMs, context engineering, reusable context packs, prompt libraries, and engineering playbooks.
- Establish standards for infrastructure automation, release automation, environment promotion, observability, data quality, access control, auditability, secure data distribution, and production readiness.
- Act as a technical bridge between global platform stakeholders, architecture, governance, security, data mastering, Databricks-aligned teams, and offshore delivery teams.
Requirements
- 10+ years of experience in data engineering, data platforms, cloud data architecture, software engineering, or related engineering domains.
- Experience operating as a senior individual contributor, technical lead, principal engineer, staff engineer, distinguished engineer, architect, or equivalent in complex data platform environments.
- Strong hands-on experience with modern data engineering, batch and/or streaming workflows, Databricks platforms, lakehouse migrations, and AWS cloud-native data engineering.
- Experience with Amazon S3, AWS Glue, AWS Lambda, AWS Lake Formation, Amazon Kinesis, AWS IAM, data lake governance, policy-based access controls, and metadata-driven controls.
- Working knowledge of Delta Lake, Apache Iceberg, Databricks Unity Catalog, metadata-driven pipelines, data catalogs, and governed lakehouse architectures.
- Experience with data mastering, MDM, reference data, market data, investment data, risk data, data quality, schema management, lineage, metadata, access control, encryption, observability, and production support.
- Practical experience using Claude, GitHub Copilot, LLMs, or similar AI-assisted development tools for code generation, refactoring, testing, documentation, debugging, pipeline analysis, SQL optimization, PySpark optimization, and production support.
- Understanding of context engineering and ability to create context packs, prompt templates, engineering playbooks, and documentation patterns for reliable AI-assisted engineering.
- Ability to evaluate AI-generated code and recommendations for correctness, security, performance, maintainability, and enterprise-standard alignment.
- Experience with design reviews, architecture discussions, technical standards, implementation planning, production readiness, distributed teams, and cross-time-zone stakeholder collaboration.
- Preferred qualifications include enterprise data cataloging, metadata management, business glossaries, semantic data layers, NeoXam DataHub, semantic modeling, Databricks certifications, AWS certifications, or equivalent practical platform experience.
Benefits
- Health and wellness coverage.
- Generous flexible time off.
- Continuous learning resources and professional development support.
- Retirement planning, continuing education, company-matched student loan contributions, and financial wellness programs.
- Family-friendly benefits, retail discounts, and referral incentive awards.
- Hyderabad-based role with a 12–9 pm IST shift and a hybrid working model of twice weekly or nine days per month in the office.
Tech Stack
DatabricksSQL
Categories
Data Engineering