Data Engineer III - Databricks, Pyspark, Python, AWS
JPMorgan Chase21 days ago
Bengaluru, IndiaSenior
Responsibilities
- Design, develop, and maintain batch and streaming big data pipelines using PySpark, Apache Spark, and Databricks.
- Own data modeling and solution design for data products, including target-state architecture, data flows, and transformation patterns.
- Build scalable ingestion and transformation workflows for high-volume datasets with strong reliability, quality, and performance.
- Develop and optimize complex SQL transformations, reconciliation queries, analytical datasets, and large-scale query workloads.
- Apply data warehousing concepts such as dimensional modeling, slowly changing dimensions, and partitioning strategies.
- Use AWS services, especially Amazon S3 and AWS data processing capabilities, for scalable storage and processing.
- Debug and troubleshoot distributed Spark workloads, including data skew, shuffle tuning, and memory or compute optimization.
- Apply modular design, efficient coding, code reviews, version control, and CI-friendly development practices.
- Partner with cross-functional stakeholders to translate requirements into big data solutions.
- Use and validate enterprise-authorized AI capabilities to support data pipeline design, documentation, test generation, and control validation while following data sensitivity, security, resiliency, and auditability requirements.
Requirements
- Formal training or certification in data engineering concepts and at least 3 years of applied experience.
- Applied data engineering or big data engineering experience with expert-level SQL, including complex joins, window functions, CTEs, optimization, and large-scale analytical problem solving.
- Production Python coding experience with the ability to write efficient and maintainable code.
- Deep Apache Spark and PySpark expertise, including performance tuning and distributed processing fundamentals.
- Strong Databricks experience for large-scale data processing and pipeline development.
- Strong understanding of data warehousing, data modeling, and scalable solution design for data platforms and products.
- Experience implementing both batch and streaming data processing solutions.
- Familiarity with AWS S3 and common AWS services used in data platforms and processing.
- Debugging, troubleshooting, problem-solving, GitHub, Bitbucket, and version control experience.
- Experience using enterprise-authorized AI capabilities in data engineering workflows, including validating AI-assisted outputs and following data handling requirements.
- Preferred experience provisioning AWS infrastructure with Infrastructure as Code tools such as Terraform or AWS CloudFormation.
Tech Stack
Categories
Data Engineering
About JPMorgan Chase
JPMorgan Chase provides consumer and commercial banking, payments, credit card, wealth management, and corporate and investment banking services to individuals, businesses, institutions, and governments. The public company (NYSE: JPM) earns revenue from interest, fees, trading, and asset management across operations in more than 100 markets. Headquartered in New York City with roots dating to 1799, it serves retail customers and prominent corporate and government clients through brands including Chase and J.P. Morgan.