
Senior Data Engineer
S&P Global2 days ago
Hyderābād, IndiaSenior
Responsibilities
- Migrate and stabilize enterprise data pipelines onto Databricks and AWS lakehouse platforms.
- Design and operate batch, streaming, event-driven, metadata-driven, and configuration-driven data pipelines.
- Implement reusable ingestion, transformation, testing, deployment, monitoring, observability, and operational support patterns.
- Apply AWS security controls, encryption, access policies, governance guardrails, lineage, metadata, and auditability practices.
- Integrate pipelines and datasets with data mastering workflows, including cleansing, enrichment, reconciliation, golden-copy generation, and downstream distribution.
- Use AI-assisted engineering tools and context-engineering practices for code generation, refactoring, testing, documentation, troubleshooting, and pipeline analysis.
- Provide senior technical guidance and contribute to architecture discussions, design reviews, code reviews, standards, and production-readiness reviews.
- Support global teams through reusable onboarding frameworks, technical documentation, runbooks, reference implementations, and engineering playbooks.
Requirements
- 6–8+ years of experience in data engineering, data platforms, cloud data architecture, software engineering, or related engineering domains.
- Experience working as a senior individual contributor, technical lead, senior data engineer, lead data platform engineer, or equivalent.
- Hands-on experience building and operating modern batch and/or streaming data pipelines.
- Experience with Databricks data platforms, lakehouse architecture, AWS-native data engineering, and production pipeline operations.
- Experience with data quality, schema management, lineage, metadata, access control, encryption, governance, observability, and reliability practices.
- Experience with data mastering, MDM, reference data, market data, investment data, risk data, or trusted data distribution workflows.
- Practical experience using Claude, GitHub Copilot, LLMs, or similar AI-assisted development platforms, including prompt and context engineering.
- Ability to evaluate and refine AI-generated code and recommendations while following security, privacy, compliance, and human-review requirements.
- Ability to provide technical guidance without direct management authority and collaborate across global, offshore, architecture, governance, security, and business-aligned teams.
- Preferred qualifications include enterprise data onboarding, semantic modeling, metadata and cataloging, NeoXam DataHub, financial-services mastering workflows, retrieval-augmented generation, embeddings, vector search, knowledge bases, relevant certifications, and regulated-environment experience.
Benefits
- Health care coverage and wellness benefits.
- Generous flexible downtime and paid time off.
- Continuous learning resources and career development support.
- Retirement planning, continuing education, company-matched student loan contributions, and financial wellness programs.
- Family-friendly benefits and perks for partners and children.
- Retail discounts and referral incentive awards.
- Benefits vary by country; additional details are provided through the company benefits site.
Tech Stack
DatabricksSQL
Categories
Data Engineering