S&P Global

Senior Data Engineer

S&P Global
Apply
2 days ago
Hyderābād, IndiaSenior

Responsibilities

  • Support migration of existing data pipelines to the target Databricks enterprise data platform and optimize them for performance, scalability, reliability, observability, security, and maintainability.
  • Design, implement, and operate reusable batch, streaming, event-driven, metadata-driven, and configuration-driven data pipeline patterns using AWS services.
  • Apply lakehouse technologies, data governance, access controls, encryption, lineage, schema management, data quality practices, and operational monitoring.
  • Integrate datasets and pipelines with enterprise data mastering workflows, including cleansing, enrichment, reconciliation, golden-copy generation, and trusted downstream distribution.
  • Use AI-assisted engineering tools and prompt/context engineering for code generation, refactoring, testing, documentation, troubleshooting, pipeline analysis, and operational support, with human review.
  • Create reusable design patterns, reference implementations, onboarding templates, pipeline frameworks, runbooks, playbooks, and technical documentation.
  • Provide senior technical guidance without direct people management and contribute to architecture discussions, design reviews, code reviews, implementation planning, and production readiness reviews.
  • Partner with global engineering, architecture, governance, security, Databricks-aligned, data mastering, and business teams to standardize platform delivery and support offshore adoption.

Requirements

  • 6–8+ years of experience in data engineering, data platforms, cloud data architecture, software engineering, or related engineering domains.
  • Demonstrated experience as a senior individual contributor, technical lead, senior data engineer, lead data platform engineer, or equivalent.
  • Strong hands-on experience with modern data engineering, batch and/or streaming workflows, Databricks platforms, lakehouse architectures, and AWS-native data engineering.
  • Experience with Amazon S3, AWS Glue, AWS Lambda, AWS Lake Formation, Amazon Kinesis, AWS IAM, and AWS monitoring, logging, and security capabilities.
  • Knowledge of Delta Lake, Apache Iceberg, Databricks Unity Catalog, metadata-driven pipelines, catalog-driven processing, and data lake governance.
  • Experience with data quality, schema management, lineage, metadata, access control, encryption, observability, alerting, telemetry, dashboards, and production support.
  • Experience with data mastering, MDM, reference data, market data, investment data, risk data, or trusted data distribution workflows.
  • Practical experience using Claude, GitHub Copilot, LLMs, prompt engineering, context engineering, and AI-assisted development workflows, including validating generated code and recommendations.
  • Ability to translate architecture into technical deliverables and reusable engineering patterns while collaborating across global and distributed teams.
  • Preferred qualifications include enterprise data onboarding, data cataloging, semantic modeling, business glossaries, lineage, NeoXam DataHub, mastering workflows, financial-services data, retrieval-augmented generation, embeddings, vector search, enterprise knowledge bases, regulated environments, and Databricks or AWS certifications.

Benefits

  • Health and wellness coverage.
  • Generous flexible downtime and time off.
  • Continuous learning resources and career development support.
  • Competitive pay, retirement planning, continuing education, company-matched student loan contributions, and financial wellness programs.
  • Family-friendly benefits and perks.
  • Retail discounts and referral incentive awards.
  • The position is located in Hyderabad, Telangana, India.

Tech Stack

DatabricksSQL

Categories

Data Engineering
Contact me