Software Engineer lll - Senior Databricks/Spark/AWS Data Engineer
JPMorgan Chase4 days ago
Responsibilities
- Design, build, maintain, review, and debug production data pipelines on Databricks using PySpark.
- Optimize PySpark jobs and Databricks clusters for performance, scalability, and cost efficiency through partitioning, caching, and resource management.
- Implement scalable end-to-end Databricks data frameworks using bronze, silver, and gold lakehouse patterns.
- Implement data quality and validation checks using Delta Lake and Delta Live Tables expectations.
- Monitor and alert on data ingestion and pipeline issues using Databricks and AWS CloudWatch.
- Automate recurring issue remediation using Databricks Workflows and AWS-native automation.
- Use AI and agentic AI solutions and AI-assisted engineering tools to improve pipeline development and code quality.
- Deliver curated, reliable datasets to BI partners using Sigma, Tableau, and Alteryx.
- Work with stakeholders to gather requirements and produce architecture and design artifacts.
- Contribute to software engineering communities of practice and explore emerging technologies.
Requirements
- Formal training or certification in software engineering concepts with at least 3 years of applied data engineering experience.
- Advanced hands-on expertise with Apache Spark, PySpark, Databricks, Delta Lake, and lakehouse patterns.
- Strong expertise in AWS data services including S3, EMR, Glue, Lambda, Athena, and related storage and compute services.
- Strong Python programming skills; Java or Scala experience is a plus.
- Experience with CI/CD pipelines and tools such as Git, Bitbucket, Jenkins, or Spinnaker.
- Hands-on experience with system design, application development, testing, operational stability, resiliency, security, and agile delivery.
- In-depth knowledge of financial-services industry IT systems.
- Solid SQL and data modeling skills; Oracle experience is a plus.
- Experience with Airflow and Autosys for workflow scheduling.
- Databricks certifications are preferred.
- Generative AI, agentic AI framework, Claude, and GitHub Copilot experience is preferred.
- Deeper expertise across the AWS cloud service catalog is preferred.
Benefits
- Competitive total rewards package with base salary, potential incentive compensation, and eligible equity programs; amounts are not specified.
- Comprehensive healthcare coverage, on-site health and wellness centers, retirement savings plan, backup childcare, tuition reimbursement, mental health support, and financial coaching.
- Equal opportunity employer with reasonable accommodation support.
Categories
Data Engineering
About JPMorgan Chase
JPMorgan Chase provides consumer and commercial banking, payments, credit card, wealth management, and corporate and investment banking services to individuals, businesses, institutions, and governments. The public company (NYSE: JPM) earns revenue from interest, fees, trading, and asset management across operations in more than 100 markets. Headquartered in New York City with roots dating to 1799, it serves retail customers and prominent corporate and government clients through brands including Chase and J.P. Morgan.