9 days ago
Pune, India or Chennai, IndiaSenior
Responsibilities
- Modernize and refactor Spark pipelines from the legacy Cloudera Hadoop platform to Databricks on AWS.
- Build and optimize Databricks-native solutions using Delta Lake, Databricks Workflows, autoscaling, and job clusters.
- Contribute to architecture, low- and mid-level designs, data models, pipeline patterns, and reusable components.
- Analyze Spark job performance and simplify complex or over-engineered pipelines into standardized patterns.
- Write clean, modular, testable code and contribute to shared frameworks, reusable libraries, and engineering standards.
- Collaborate with architects, platform teams, and DevOps engineers on technical decisions, troubleshooting, and implementation guidance.
- Develop unit, integration, and data validation tests and support production releases and post-deployment validation.
Requirements
- At least 8 years of experience in data engineering or distributed systems.
- Strong expertise in Apache Spark, including Scala, Java, or PySpark, as well as Databricks on AWS and Delta Lake.
- Experience with SQL, AWS services, and large-scale distributed data processing.
- Experience modernizing or refactoring legacy data platforms into cloud-based architectures.
- Strong background in Spark performance tuning and large-scale batch optimization.
- Ability to translate architecture into implementable designs and understand data modeling and pipeline orchestration patterns.
- Strong problem-solving, communication, accountability, collaboration, and time-bound delivery skills.
- Bachelor’s degree or university degree and equivalent experience.
Benefits
- Full-time employment.
Tech Stack
Categories
Data Engineering
About Citi
Citi is a public financial-services company offering consumer and institutional banking, credit cards, wealth management, treasury and trade solutions, and capital-markets services. It serves individuals, corporations, financial institutions, and governments in more than 160 countries and jurisdictions, earning interest and fee income from lending, payments, trading, and advisory. Founded in 1812 and headquartered in New York, it trades on the NYSE under the ticker C.