Lead Software Engineer
JPMorgan Chase3 days ago
Responsibilities
- Lead the architecture and delivery of high-throughput, low-latency data pipelines on Databricks using Apache Spark.
- Establish Lakehouse patterns with Delta Lake, including ACID transactions, schema evolution, time travel, Z-ordering, and compaction.
- Lead adoption of enterprise-authorized AI-assisted engineering practices and establish standards for validating AI-generated outputs.
- Own Databricks cluster strategy and configuration, including runtime selection, autoscaling, Spark settings, cluster policies, pools, and instance profiles.
- Orchestrate Databricks Workflows and integrate pipelines with AWS eventing and orchestration services.
- Design secure ingestion and transformation frameworks using Databricks, Delta tables, unmanaged tables, and Airflow DAGs.
- Enforce data quality, lineage, and governance with Unity Catalog and AWS Glue Catalog.
- Tune Spark and Databricks workloads for performance, scalability, cost, and runtime efficiency.
- Build reusable libraries, frameworks, and APIs in Python and/or Java with unit, integration, and data validation testing.
- Implement CI/CD for data projects using Git workflows, Terraform infrastructure deployments, automated releases, code reviews, and engineering standards.
- Provide technical mentorship and promote secure coding, testing, observability, reliability, and reuse of effective engineering patterns.
Requirements
- Formal training or certification in software engineering concepts and 5+ years of applied experience.
- Advanced software engineering and data engineering experience, including significant production delivery with Apache Spark on Databricks and/or AWS EMR.
- Experience leading approved AI-assisted software development tools for coding, code review, test acceleration, and troubleshooting.
- Understanding of responsible AI use, data sensitivity, secure input and output handling, resiliency, security, and compliant team adoption.
- Advanced hands-on Databricks experience with Delta Lake, Unity Catalog, Workflows, Repos/notebooks, SQL Warehouses, cluster configuration, and optimization.
- Experience architecting, building, and operating reliable batch and streaming ETL/ELT pipelines with schema evolution, SLAs, and reliability practices.
- Deep Spark performance tuning and bottleneck diagnosis experience.
- Strong Python programming proficiency for data processing, platform tooling, and automation.
- Strong SQL, analytics data modeling, dimensional/star schema, and Lakehouse expertise.
- Experience with secure handling, roles and instance profiles, secrets management, encryption at rest and in transit, network controls, and security expectations.
- Preferred: Delta Live Tables, advanced Databricks governance, AWS networking, Terraform, data-system observability, cost optimization, Airflow, Genie, Streamlit, React, CI/CD, testing strategy, and technical mentorship.
Benefits
- Competitive total rewards package with role-based base salary, eligible commission-based pay, and/or discretionary incentive compensation.
- Comprehensive health care coverage and on-site health and wellness centers.
- Retirement savings plan and backup childcare.
- Tuition reimbursement, mental health support, and financial coaching.
- Equal opportunity employment and reasonable accommodations for religious practices, mental health needs, and physical disabilities.
Categories
Data Engineering
About JPMorgan Chase
JPMorgan Chase provides consumer and commercial banking, payments, credit card, wealth management, and corporate and investment banking services to individuals, businesses, institutions, and governments. The public company (NYSE: JPM) earns revenue from interest, fees, trading, and asset management across operations in more than 100 markets. Headquartered in New York City with roots dating to 1799, it serves retail customers and prominent corporate and government clients through brands including Chase and J.P. Morgan.