Lead Software Engineer - DataBricks, Spark, Terraform
JPMorgan Chase15 hours ago
Base Salary
$138k - $185k/yr
Responsibilities
- Define and drive the technical roadmap, architecture, standards, and reference patterns for a Databricks lakehouse and database platform.
- Design curated datasets using Delta Lake, medallion architecture, dimensional or semantic modeling, and consistent partitioning and performance practices.
- Build and operate batch and streaming ingestion, transformation frameworks, orchestration, and curated data layers.
- Establish availability, SLOs, runbooks, alerting, incident response, observability, data quality checks, and operational hygiene for pipelines and SQL workloads.
- Implement governance, least-privilege access, auditing, data classification, lifecycle management, and Unity Catalog controls.
- Tune Spark and SQL workloads, optimize clusters and warehouses, manage caching and storage patterns, and implement cost observability or chargeback as needed.
- Set standards for code quality, testing, CI/CD, branching, documentation, code reviews, reusable libraries, and templates.
- Coach engineers, lead design reviews, and partner with stakeholders to translate business needs into scalable data-platform capabilities.
- Drive team adoption of enterprise-authorized AI-assisted development and automation tools while establishing validation standards for correctness, performance, security, and responsible data handling.
Requirements
- At least 7 years of experience in data engineering or platform engineering, including hands-on production Databricks experience.
- Strong proficiency with Spark, PySpark or Scala, and SQL, including performance tuning and troubleshooting.
- Experience building and operating data platforms covering batch and streaming ingestion, transformation, orchestration, and curated data layers.
- Experience with Data Lake table design and optimization using Databricks, Snowflake, or AWS, including partitioning, Z-ORDER, and file sizing.
- Familiarity with Unity Catalog or equivalent governance tooling, including permissions, catalogs, schemas, lineage, and auditing concepts.
- Strong software engineering fundamentals including Git, automated testing, code reviews, modular design, and documentation.
- Experience implementing logs, metrics, traces, data quality checks, and monitoring for pipelines and SQL workloads.
- Strong communication skills and the ability to lead technical decisions across teams.
- Demonstrated experience leading effective use of approved AI-assisted software development tools for coding, code review, test acceleration, or troubleshooting.
- Understanding of responsible AI use in engineering workflows, data sensitivity, secure input and output handling, resiliency, security, and compliant adoption practices.
- Preferred experience with Databricks Workflows, Delta Live Tables, Structured Streaming, Databricks SQL Warehouses, dbt, Terraform, regulated-data environments, privacy controls, and enterprise data governance programs.
Benefits
- Competitive total rewards package with base salary determined by role, experience, skill set, and location; eligible roles may also receive commission-based pay or discretionary incentive compensation.
- Comprehensive health care coverage and on-site health and wellness centers.
- Retirement savings plan, backup childcare, tuition reimbursement, mental health support, and financial coaching.
- Additional compensation and benefits details are provided during the hiring process.
Tech Stack
Categories
Data Engineering
About JPMorgan Chase
JPMorgan Chase provides consumer and commercial banking, payments, credit card, wealth management, and corporate and investment banking services to individuals, businesses, institutions, and governments. The public company (NYSE: JPM) earns revenue from interest, fees, trading, and asset management across operations in more than 100 markets. Headquartered in New York City with roots dating to 1799, it serves retail customers and prominent corporate and government clients through brands including Chase and J.P. Morgan.