14 days ago
Pune, IndiaStaff+
Responsibilities
- Modernize and refactor Spark pipelines from Cloudera Hadoop to Databricks on AWS.
- Build and optimize Databricks-native solutions using Delta Lake, Databricks Workflows, autoscaling, and job clusters.
- Translate high-level architecture into detailed designs and define data models, pipeline patterns, and reusable components.
- Optimize Spark job performance and simplify complex or over-engineered data pipelines.
- Develop clean, modular, testable code and contribute to shared frameworks, reusable libraries, and engineering standards.
- Provide technical guidance and troubleshooting support while collaborating with architects, platform teams, DevOps engineers, and global stakeholders.
- Develop unit, integration, and data validation tests and support production releases and post-deployment validation.
Requirements
- 10+ years of experience in data engineering or distributed systems.
- Strong expertise in Apache Spark, JavaSpark or PySpark, Databricks on AWS, and Delta Lake.
- Experience with SQL, AWS services, and large-scale distributed data processing.
- Experience modernizing or refactoring legacy data platforms into cloud-based architectures.
- Strong background in Spark performance tuning and large-scale batch optimization.
- Ability to translate architecture into implementable designs and understanding of data modeling and pipeline orchestration patterns.
- Bachelor’s degree or university degree, or equivalent experience.
- Strong problem-solving, communication, collaboration, accountability, and decision-making skills.
Tech Stack
Categories
Data Engineering
About Citi
Citi's mission is to serve as a trusted partner to our clients by responsibly providing financial services that enable growth and economic progress. Our core activities are safeguarding assets, lending money, making payments and accessing the capital markets on behalf of our clients. We have over 200 years of experience helping our clients meet the world's toughest challenges and embrace its greatest opportunities. We are Citi, the global bank – an institution connecting millions of people across hundreds of countries and cities. For information on Citi’s commitment to privacy, visit on.citi/privacy.