Associate Director, Data Platforms — Technical Lead
S&P Global9 days ago
Hyderābād, IndiaStaff+
Responsibilities
- Plan and execute the transition of existing data pipelines to the target Databricks data platform.
- Implement repeatable engineering patterns for ingestion, transformation, testing, deployment, monitoring, and dataset onboarding.
- Design and operate cloud-native batch and streaming data pipelines using AWS services.
- Support lakehouse architecture, governed data access, metadata-driven pipelines, and enterprise data platform standards.
- Integrate platform pipelines and datasets with data mastering capabilities, including cleansing, enrichment, reconciliation, golden-copy generation, and downstream distribution.
- Establish data quality controls, schema and metadata alignment, stewardship workflows, and reconciliation processes.
- Create reusable design patterns, reference implementations, onboarding templates, pipeline frameworks, documentation, and operational runbooks.
- Provide senior technical individual-contributor leadership through architecture discussions, design reviews, implementation planning, code reviews, technical standards, and production-readiness reviews.
- Establish infrastructure automation, release management, environment promotion, monitoring, alerting, telemetry, and incident-management practices.
- Partner with platform, architecture, governance, security, Databricks-aligned, data mastering, and globally distributed teams.
Requirements
- 8+ years of experience in data engineering, data platforms, cloud data architecture, or related engineering domains.
- Proven ability to drive technical initiatives and influence engineering outcomes in complex execution-focused environments.
- Experience collaborating with global or distributed teams across time zones.
- Strong hands-on experience with modern data engineering, pipeline development, and batch or streaming workflows.
- Experience with Databricks data platforms and migration to lakehouse-oriented architectures.
- Strong AWS cloud-native data engineering experience, including S3, Glue, Lambda, Lake Formation, Kinesis, IAM, and AWS monitoring, logging, and security capabilities.
- Working knowledge of Delta Lake, Apache Iceberg, Databricks Unity Catalog, metadata-driven pipelines, data lake governance, lakehouse architecture, and catalog-driven processing.
- Experience with data mastering, MDM, reference data, market data, investment data, risk data, or trusted data distribution workflows.
- Understanding of data quality, schema management, lineage, metadata, access control, encryption, and production pipeline operations.
- Experience designing or implementing observability and reliability practices, including monitoring, alerting, telemetry, dashboards, and production support procedures.
- Practical experience using Claude, large language models, or similar AI-assisted development platforms for code generation, refactoring, testing, documentation, debugging, troubleshooting, and pipeline analysis.
- Ability to evaluate and validate AI-generated code or recommendations before applying them to production engineering work.
- Strong communication and stakeholder-management skills, with the ability to translate architecture into actionable technical deliverables and reusable engineering patterns.
Benefits
- Health care coverage and wellness benefits.
- Generous flexible downtime and paid time off.
- Continuous-learning resources and career development opportunities.
- Competitive pay, retirement planning, continuing education, company-matched student loan contributions, and financial wellness programs.
- Family-friendly benefits and perks for partners and children.
- Retail discounts and referral incentive awards.
- Based in Hyderabad with a 12–9 pm IST shift and a hybrid schedule requiring office work twice per week or nine days per month.
Tech Stack
DatabricksSQL
Categories
Data Engineering