Associate Director, Data Platforms — Technical Lead
S&P Global9 days ago
Hyderābād, IndiaStaff+
Responsibilities
- Plan and execute the transition of existing data pipelines to the target Databricks data platform.
- Implement repeatable patterns for data ingestion, transformation, testing, deployment, monitoring, and production support.
- Design and operate cloud-native batch and streaming data pipelines using AWS services and lakehouse technologies.
- Define reusable, metadata-driven onboarding patterns, templates, frameworks, reference implementations, documentation, and runbooks.
- Integrate platform pipelines and datasets with enterprise data mastering capabilities, including reconciliation, golden-copy generation, and downstream distribution.
- Establish data quality, schema, lineage, metadata, access-control, encryption, auditability, and secure-distribution controls.
- Provide senior technical individual-contributor leadership through architecture discussions, design reviews, code reviews, implementation planning, and production-readiness reviews.
- Establish infrastructure automation, release management, environment promotion, observability, incident management, and operational support practices.
- Collaborate with platform, architecture, governance, security, Databricks-aligned, data mastering, and globally distributed teams.
- Apply AI-assisted engineering workflows to code generation, refactoring, testing, documentation, debugging, pipeline migration, and data quality analysis.
Requirements
- At least 8 years of experience in data engineering, data platforms, cloud data architecture, or related engineering domains.
- Proven ability to drive technical initiatives and influence engineering outcomes in complex execution-focused environments.
- Hands-on experience with modern data engineering, including batch and/or streaming workflows.
- Experience with Databricks-based data platforms and migrating pipelines to lakehouse-oriented architectures.
- Strong familiarity with AWS data engineering services including Amazon S3, AWS Glue, AWS Lambda, AWS Lake Formation, Amazon Kinesis, and AWS IAM.
- Working knowledge of Delta Lake, Apache Iceberg, Databricks Unity Catalog, metadata-driven pipelines, data lake governance, and lakehouse architecture.
- Experience with data mastering, MDM, reference data, market data, investment data, risk data, or trusted data distribution workflows.
- Understanding of data quality, schema management, lineage, metadata, access control, encryption, observability, reliability, and production support.
- Familiarity with version control, automated testing, CI/CD, infrastructure automation, release management, monitoring, and operational dashboards.
- Practical experience using Claude, large language models, or similar AI-assisted development platforms in engineering workflows.
- Ability to evaluate and validate AI-generated code or recommendations before production use.
- Strong communication, stakeholder management, and collaboration skills across US-based, offshore, and global teams.
- Ability to translate architecture into actionable technical deliverables and reusable engineering patterns.
Benefits
- Health care coverage and wellness benefits.
- Generous flexible time off.
- Continuous learning and career-development resources.
- Retirement planning, continuing education support, company-matched student loan contributions, and financial wellness programs.
- Family-friendly benefits and perks.
- Retail discounts and referral incentive awards.
- Hyderabad-based role with a working model of working from the office twice a week or 9 days per month.
- Shift time is 12:00 to 21:00 IST.
Tech Stack
DatabricksSQL
Categories
Data Engineering