Lead Software Engineer - Data Engg. - Databricks / Snowflake
JPMorgan Chase7 days ago
Responsibilities
- Design and develop scalable software solutions and an enterprise control plane for data pipeline definition, scheduling, deployment, governance, and runtime management.
- Build self-service APIs, SDKs, templates, configuration-driven onboarding, pipeline metadata management, and standardized engineering guardrails.
- Develop and maintain data pipelines, processing workflows, and fact/dimension models using Python, PySpark, SQL, Databricks, and AWS.
- Implement data quality, security, lineage, observability, dashboards, alerting, runbooks, retries, backfills, and automated remediation.
- Participate in the full software development lifecycle and provide production support for pipeline and platform services.
- Collaborate with stakeholders on data management strategy, compliant solutions, documentation, and transformation rules.
- Mentor engineers, lead communities of practice, and drive adoption of modern and responsible AI-assisted engineering practices.
Requirements
- Formal training or certification in software engineering concepts and at least 5 years of applied experience.
- Experience with large-scale data management and ETL/ELT, including strong SQL, Python, PySpark, performance tuning, and query optimization.
- Hands-on experience with Databricks, Spark, cloud data lake patterns, and AWS service integrations.
- Experience building platform services, control planes, orchestration platforms, APIs, configuration-driven systems, and version-compatible services.
- Understanding of data quality, security by design, lineage, auditability, IAM, least privilege, and secrets management.
- Production engineering experience with observability, monitoring, alerting, incident response, and operational excellence for always-on services.
- Proficiency with CI/CD and release engineering, including quality gates, automated testing, safe deployments, and rollbacks using tools such as Jenkins/Jules, Spinnaker, and Sonar.
- Experience leading the effective and secure use of approved AI-assisted software development tools and validating their outputs.
- Preferred experience with Databricks Workflows/Jobs, Airflow, Step Functions, DAGs, replays, backfills, Unity Catalog, Terraform, CloudFormation, internal CLIs, templates, FinOps, Spark/Databricks cost controls, and data formats such as Parquet, JSON, CSV, Avro, and Delta Lake.
- Databricks or AWS certifications and experience with regulatory reporting and financial data aggregation are preferred.
Benefits
- Competitive total rewards package with base salary determined by role, experience, skill set, and location; eligible roles may also receive incentive compensation.
- Comprehensive health care coverage, on-site health and wellness centers, retirement savings plan, backup childcare, tuition reimbursement, mental health support, and financial coaching.
- Equal opportunity employer offering reasonable accommodations for religious practices, mental health, and physical disability needs.
- Enhanced pre-hire screening, including criminal and credit background checks where permitted by law, applies to designated high-risk technology roles.
Tech Stack
Categories
Data EngineeringDevOps
About JPMorgan Chase
JPMorgan Chase provides consumer and commercial banking, payments, credit card, wealth management, and corporate and investment banking services to individuals, businesses, institutions, and governments. The public company (NYSE: JPM) earns revenue from interest, fees, trading, and asset management across operations in more than 100 markets. Headquartered in New York City with roots dating to 1799, it serves retail customers and prominent corporate and government clients through brands including Chase and J.P. Morgan.