5 months ago
Pune, IndiaStaff+
Responsibilities
- Design and implement scalable, fault-tolerant batch and real-time data processing pipelines.
- Develop performant data models, schema designs, and cloud-optimized data formats.
- Lead migration of legacy SQL Server platforms, stored procedures, and ETL jobs to modern Data Lakehouse and distributed processing environments.
- Build automated data parity testing, validation, schema evolution, phased cutover, and shadow execution frameworks.
- Establish performance benchmarks and optimize SQL queries and distributed computing clusters for latency and cost.
- Write high-performance Python and maintain CI/CD pipelines for data-job testing and deployment.
- Monitor system health, troubleshoot data-lifecycle bottlenecks, and implement data integrity, validation, and encryption controls.
- Mentor junior and mid-level engineers, conduct code reviews, allocate work, and advise developers and analysts.
- Translate business requirements into technical specifications and collaborate with Product Managers and management teams on data availability and system enhancements.
- Assess and manage operational, compliance, security, and control risks.
Requirements
- 12+ years of experience in software building and platform engineering.
- Extensive Data Engineering experience focused on Big Data ecosystems and high-volume data processing pipelines.
- Expert-level hands-on Python experience and strong SQL query-writing skills.
- Hands-on experience with Hadoop, YARN, Hive, Impala, Spark, Spark SQL, Kafka, NoSQL databases, Trino, Presto, or Starburst.
- Experience with MongoDB, HBase, Avro, Parquet, CSV, and JSON data formats.
- Experience with Unix-based operating systems, shell scripting, and source code management tools such as Bitbucket and Git.
- Strong computer science fundamentals in data structures, algorithms, databases, and operating systems.
- Ability to reverse-engineer legacy SQL and scripts, document business logic, and manage data lineage.
- Experience with data lineage tools such as Collibra or Informatica.
- Experience managing technical change when migrating from legacy BI tools to modern query engines.
- Strong problem-solving, communication, automation, and stakeholder collaboration skills.
- Familiarity with AI tools is preferred.
Tech Stack
Apache FlinkApache HadoopApache HBaseApache HiveApache KafkaApache SparkGitInformaticaMicrosoft SQL ServerMongoDBPrestoPythonSQLYarn
Categories
Data Engineering
About Citi
Citi is a public financial-services company offering consumer and institutional banking, credit cards, wealth management, treasury and trade solutions, and capital-markets services. It serves individuals, corporations, financial institutions, and governments in more than 160 countries and jurisdictions, earning interest and fee income from lending, payments, trading, and advisory. Founded in 1812 and headquartered in New York, it trades on the NYSE under the ticker C.