2 months ago
Pune, IndiaStaff+
Responsibilities
- Design, develop, and deploy scalable batch and real-time end-to-end ETL/ELT data pipelines.
- Deploy and maintain Databricks workspaces on AWS or GCP and integrate cloud storage, IAM, secrets management, and serverless query engines.
- Diagnose and resolve performance bottlenecks in Spark clusters, SQL queries, and Databricks jobs.
- Optimize Delta Lake storage layouts through partitioning, Z-Ordering, and vacuuming.
- Implement automated data validation, data quality monitoring, metadata management, and governance controls using Databricks Unity Catalog.
- Ensure data engineering practices support internal governance policies and financial regulations such as BCBS 239.
- Lead peer code reviews, enforce coding standards, and mentor junior and mid-level data engineers.
- Establish and maintain CI/CD pipelines for deploying dbt packages, Spark jobs, and Databricks workflows.
- Partner with data science teams to operationalize machine learning models.
- Collaborate with business intelligence developers to build efficient semantic layers for reporting.
- Act as a subject matter expert and technical lead for pipeline architecture, distributed computing, and cloud-native integrations.
Requirements
- Production-grade proficiency in Python, including standard libraries, pandas, and pytest or comparable testing frameworks.
- Advanced SQL skills, including window functions, common table expressions, and query optimization.
- Deep hands-on experience with Apache Spark and PySpark for processing multi-terabyte datasets in distributed clusters.
- At least 3 years of hands-on experience developing with Databricks.
- Expertise in Delta Lake ACID transactions, Delta Live Tables, Unity Catalog, and Databricks Workflows.
- Extensive experience deploying Databricks on either AWS or GCP.
- Proficiency with cloud-native services including AWS S3, EC2, IAM, EMR, Athena, and Redshift, or GCP GCS, Compute Engine, IAM, Dataproc, and BigQuery.
- Understanding of data warehousing, dimensional modeling, star and snowflake schemas, slowly changing dimensions, and medallion architecture.
- Strong analytical, troubleshooting, communication, collaboration, and prioritization skills.
- Ability to explain complex technical architectures to non-technical stakeholders and work in a global, cross-functional engineering environment.
- Databricks Certified Data Engineer Professional or Databricks Certified Associate Developer for Apache Spark certification is preferred.
- AWS Certified Solutions Architect, AWS Certified Data Engineer, or Google Cloud Professional Data Engineer certification is preferred.
Benefits
- Full-time employment.
Tech Stack
Amazon RedshiftApache SparkAWSDatabricksdbtGitHub ActionsGoogle BigQueryGoogle Cloud PlatformJenkinsPandaspytestPythonSQLVault
Categories
Data Engineering
About Citi
Citi is a public financial-services company offering consumer and institutional banking, credit cards, wealth management, treasury and trade solutions, and capital-markets services. It serves individuals, corporations, financial institutions, and governments in more than 160 countries and jurisdictions, earning interest and fee income from lending, payments, trading, and advisory. Founded in 1812 and headquartered in New York, it trades on the NYSE under the ticker C.