2 months ago
Pune, IndiaStaff+
Responsibilities
- Modernize and refactor Spark pipelines from Cloudera Hadoop to Databricks on AWS.
- Build and optimize Databricks-native solutions using Delta Lake, Databricks Workflows, autoscaling, and job clusters.
- Translate high-level architecture into detailed designs and define data models, pipeline patterns, and reusable components.
- Optimize Spark job performance and simplify complex or over-engineered data pipelines.
- Develop clean, modular, testable code and contribute to shared frameworks, reusable libraries, and engineering standards.
- Provide technical guidance and troubleshooting support while collaborating with architects, platform teams, DevOps engineers, and global stakeholders.
- Develop unit, integration, and data validation tests and support production releases and post-deployment validation.
Requirements
- 10+ years of experience in data engineering or distributed systems.
- Strong expertise in Apache Spark, JavaSpark or PySpark, Databricks on AWS, and Delta Lake.
- Experience with SQL, AWS services, and large-scale distributed data processing.
- Experience modernizing or refactoring legacy data platforms into cloud-based architectures.
- Strong background in Spark performance tuning and large-scale batch optimization.
- Ability to translate architecture into implementable designs and understanding of data modeling and pipeline orchestration patterns.
- Bachelor’s degree or university degree, or equivalent experience.
- Strong problem-solving, communication, collaboration, accountability, and decision-making skills.
Tech Stack
Categories
Data Engineering
About Citi
Citi is a public financial-services company offering consumer and institutional banking, credit cards, wealth management, treasury and trade solutions, and capital-markets services. It serves individuals, corporations, financial institutions, and governments in more than 160 countries and jurisdictions, earning interest and fee income from lending, payments, trading, and advisory. Founded in 1812 and headquartered in New York, it trades on the NYSE under the ticker C.