Distinguished Engineer, Data Platforms
S&P Global4 days ago
Hyderābād, IndiaStaff+
Responsibilities
- Lead the transition of existing data pipelines to the target Databricks-based enterprise data platform.
- Drive reusable ingestion, transformation, testing, deployment, monitoring, and onboarding patterns for batch and streaming data.
- Guide AWS cloud-native data platform architecture using services including Amazon S3, AWS Glue, AWS Lambda, Amazon Kinesis, and AWS Lake Formation.
- Influence lakehouse, Delta Lake, Apache Iceberg, Unity Catalog, metadata-driven processing, governance, and semantic modeling decisions.
- Support data mastering integrations, data quality controls, reconciliation, metadata alignment, stewardship, and trusted downstream distribution.
- Create context packs, prompt libraries, and AI-enabled playbooks using Claude, GitHub Copilot, and LLMs for data engineering workflows.
- Provide technical direction, design review, code review, implementation planning, and mentorship to distributed engineering teams without direct people management.
- Establish standards for security, lineage, schema consistency, observability, infrastructure automation, release management, and production support.
Requirements
- At least 10 years of experience in data engineering, data platforms, cloud data architecture, software engineering, or related engineering domains.
- Experience operating as a senior individual contributor, technical lead, principal, staff, distinguished engineer, architect, or equivalent.
- Strong hands-on experience with modern batch and/or streaming data engineering and Databricks-based lakehouse migrations.
- Strong AWS data engineering experience with Amazon S3, AWS Glue, AWS Lambda, AWS Lake Formation, Amazon Kinesis, security, orchestration, and monitoring capabilities.
- Knowledge of Delta Lake, Apache Iceberg, Databricks Unity Catalog, metadata-driven pipelines, data catalogs, and governed lakehouse architectures.
- Experience with data mastering, MDM, reference data, market data, investment data, risk data, trusted data distribution, data quality, lineage, metadata, access control, and encryption.
- Experience with observability and reliability practices including alerting, monitoring, telemetry, dashboards, and production support.
- Practical experience using Claude, GitHub Copilot, LLMs, or similar AI-assisted engineering tools for code generation, refactoring, testing, documentation, debugging, SQL optimization, PySpark optimization, or pipeline analysis.
- Understanding of context engineering and the ability to create reusable context packs, prompt templates, engineering playbooks, and documentation patterns.
- Ability to influence architecture and technical direction across distributed global teams without formal people-management authority.
- Preferred qualifications include enterprise cataloging, metadata management, business glossaries, semantic modeling, NeoXam DataHub, Databricks certifications, AWS certifications, or equivalent practical experience.
Benefits
- Health care coverage and wellness benefits.
- Generous flexible time off.
- Continuous learning resources and career development support.
- Retirement planning, continuing education, company-matched student loan contributions, and financial wellness programs.
- Family-friendly benefits, retail discounts, and referral incentive awards.
- Hyderabad-based role with a 12–9 pm IST shift and a hybrid schedule requiring work from the office twice weekly or nine days per month.
Tech Stack
DatabricksSQL
Categories
Data Engineering