4 days ago
Pune, India or Chennai, IndiaSenior
Responsibilities
- Modernize and refactor Spark pipelines while transitioning from Cloudera Hadoop to Databricks on AWS.
- Build and optimize Databricks solutions using Delta Lake, Databricks Workflows, autoscaling, and job clusters.
- Translate high-level architecture into detailed technical designs, data models, pipeline patterns, and reusable components.
- Analyze and improve Spark job performance and simplify complex data-processing pipelines.
- Develop clean, modular, testable code and contribute to shared frameworks, reusable libraries, and engineering standards.
- Collaborate with architects, platform teams, and DevOps engineers on technical decisions, troubleshooting, and implementation guidance.
- Develop unit, integration, and data-validation tests and support production releases and post-deployment validation.
Requirements
- 8+ years of experience in data engineering or distributed systems.
- Strong expertise in Apache Spark, including ScalaSpark, JavaSpark, or PySpark, Databricks on AWS, and Delta Lake.
- Experience with SQL, AWS services, and large-scale distributed data processing.
- Experience modernizing or refactoring legacy data platforms into cloud-based architectures.
- Strong background in Spark performance tuning and large-scale batch optimization.
- Ability to translate architecture into implementable designs and understanding of data modeling and pipeline orchestration patterns.
- Bachelor’s degree or university degree and equivalent experience.
- Strong problem-solving, communication, collaboration, accountability, and prioritization skills.
Benefits
- Full-time position.
- Equal opportunity employer with consideration for qualified candidates regardless of legally protected characteristics.
- Reasonable accommodations are available for applicants with disabilities.
Tech Stack
Categories
Data Engineering
About Citi
Citi is a public financial-services company offering consumer and institutional banking, credit cards, wealth management, treasury and trade solutions, and capital-markets services. It serves individuals, corporations, financial institutions, and governments in more than 160 countries and jurisdictions, earning interest and fee income from lending, payments, trading, and advisory. Founded in 1812 and headquartered in New York, it trades on the NYSE under the ticker C.