
Senior Data Engineer
S&P Global2 days ago
Hyderābād, IndiaSenior
Responsibilities
- Support migration of existing data pipelines to the target Databricks enterprise data platform and optimize them for performance, scalability, reliability, observability, security, and maintainability.
- Design, implement, and operate reusable batch, streaming, event-driven, metadata-driven, and configuration-driven data pipeline patterns using AWS services.
- Apply lakehouse technologies, data governance, access controls, encryption, lineage, schema management, data quality practices, and operational monitoring.
- Integrate datasets and pipelines with enterprise data mastering workflows, including cleansing, enrichment, reconciliation, golden-copy generation, and trusted downstream distribution.
- Use AI-assisted engineering tools and prompt/context engineering for code generation, refactoring, testing, documentation, troubleshooting, pipeline analysis, and operational support, with human review.
- Create reusable design patterns, reference implementations, onboarding templates, pipeline frameworks, runbooks, playbooks, and technical documentation.
- Provide senior technical guidance without direct people management and contribute to architecture discussions, design reviews, code reviews, implementation planning, and production readiness reviews.
- Partner with global engineering, architecture, governance, security, Databricks-aligned, data mastering, and business teams to standardize platform delivery and support offshore adoption.
Requirements
- 6–8+ years of experience in data engineering, data platforms, cloud data architecture, software engineering, or related engineering domains.
- Demonstrated experience as a senior individual contributor, technical lead, senior data engineer, lead data platform engineer, or equivalent.
- Strong hands-on experience with modern data engineering, batch and/or streaming workflows, Databricks platforms, lakehouse architectures, and AWS-native data engineering.
- Experience with Amazon S3, AWS Glue, AWS Lambda, AWS Lake Formation, Amazon Kinesis, AWS IAM, and AWS monitoring, logging, and security capabilities.
- Knowledge of Delta Lake, Apache Iceberg, Databricks Unity Catalog, metadata-driven pipelines, catalog-driven processing, and data lake governance.
- Experience with data quality, schema management, lineage, metadata, access control, encryption, observability, alerting, telemetry, dashboards, and production support.
- Experience with data mastering, MDM, reference data, market data, investment data, risk data, or trusted data distribution workflows.
- Practical experience using Claude, GitHub Copilot, LLMs, prompt engineering, context engineering, and AI-assisted development workflows, including validating generated code and recommendations.
- Ability to translate architecture into technical deliverables and reusable engineering patterns while collaborating across global and distributed teams.
- Preferred qualifications include enterprise data onboarding, data cataloging, semantic modeling, business glossaries, lineage, NeoXam DataHub, mastering workflows, financial-services data, retrieval-augmented generation, embeddings, vector search, enterprise knowledge bases, regulated environments, and Databricks or AWS certifications.
Benefits
- Health and wellness coverage.
- Generous flexible downtime and time off.
- Continuous learning resources and career development support.
- Competitive pay, retirement planning, continuing education, company-matched student loan contributions, and financial wellness programs.
- Family-friendly benefits and perks.
- Retail discounts and referral incentive awards.
- The position is located in Hyderabad, Telangana, India.
Tech Stack
DatabricksSQL
Categories
Data Engineering