Director, Distinguished Engineer - Data Platforms
S&P Global21 days ago
Hyderābād, IndiaStaff+
Responsibilities
- Plan and execute the transition of existing data pipelines to the target enterprise data platform in partnership with the Databricks team.
- Define reusable ingestion, transformation, testing, deployment, monitoring, onboarding, and governance patterns for batch and streaming data workflows.
- Guide cloud-native AWS data pipeline design using services including Amazon S3, AWS Glue, AWS Lambda, Amazon Kinesis, and AWS Lake Formation.
- Influence architecture involving Databricks, Delta Lake, Apache Iceberg, Unity Catalog, metadata-driven processing, and governed lakehouse design.
- Create AI-assisted engineering practices, context packs, prompt libraries, and playbooks for pipeline migration, code generation, testing, code review, and troubleshooting.
- Support data mastering, reconciliation, metadata alignment, stewardship, semantic modeling, data catalogs, lineage, and trusted downstream data distribution.
- Provide technical direction, design and code review guidance, implementation patterns, and production-readiness support to distributed engineering teams.
- Help establish infrastructure automation, release automation, environment promotion, observability, reliability, incident management, and responsible AI controls.
Requirements
- 10+ years of experience in data engineering, data platforms, cloud data architecture, software engineering, or related engineering domains.
- Experience operating as a senior individual contributor, technical lead, principal engineer, staff engineer, distinguished engineer, architect, or equivalent in complex data-platform environments.
- Strong hands-on experience building modern batch and/or streaming data pipelines and working with Databricks-based lakehouse platforms.
- Strong AWS cloud-native data-engineering experience, including Amazon S3, AWS Glue, AWS Lambda, AWS Lake Formation, Amazon Kinesis, security, orchestration, and monitoring capabilities.
- Working knowledge of Delta Lake, Apache Iceberg, Databricks Unity Catalog, metadata-driven pipelines, data catalogs, and governed lakehouse architectures.
- Experience with data integration involving mastering, MDM, reference data, market data, investment data, risk data, or trusted data distribution workflows.
- Experience with data quality, schema management, lineage, metadata, access control, encryption, observability, alerting, monitoring, telemetry, and production support.
- Experience with AWS IAM, policy-based access controls, version control, automated testing, CI/CD, infrastructure automation, release management, and monitoring.
- Practical experience using Claude, GitHub Copilot, LLMs, or similar AI-assisted development tools for engineering or data-engineering workflows.
- Experience applying LLMs to code generation, refactoring, test generation, documentation, debugging, pipeline analysis, SQL optimization, PySpark optimization, or production support.
- Understanding of context engineering and ability to create reusable context packs, prompt templates, engineering playbooks, and documentation patterns.
- Ability to evaluate AI-generated outputs for correctness, security, performance, maintainability, and enterprise-standard alignment.
- Strong communication and stakeholder-engagement skills across offshore, global, architecture, governance, security, and platform teams.
- Preferred qualifications include enterprise data cataloging, metadata management, semantic modeling, NeoXam DataHub, mastering workflows, Databricks certifications, AWS certifications, or equivalent practical experience.
Benefits
- Health and wellness coverage.
- Generous flexible downtime and paid time off.
- Continuous-learning resources and professional development opportunities.
- Retirement planning, continuing education, company-matched student loan contributions, and financial wellness programs.
- Family-friendly benefits, retail discounts, and referral incentive awards.
- Hyderabad-based role with a 12:00 p.m. to 9:00 p.m. IST shift and a hybrid schedule requiring work from the office twice weekly or nine days per month.
Tech Stack
DatabricksSQL
Categories
Data Engineering