Software Engineer III - Python, Databricks and AWS
JPMorgan Chase1 day ago
Responsibilities
- Implement resilient batch and streaming ingestion pipelines from AWS sources into Databricks, including CDC where needed.
- Build maintainable pipelines using Delta Live Tables or Databricks Jobs with modular structure, testing, and documentation.
- Productionize workloads with Databricks Workflows and Jobs, retries, checkpointing, idempotency, and safe reruns.
- Apply data governance and security controls including least privilege, PII classification, auditing, lineage, metadata, and controlled sharing.
- Tune Spark and Delta workloads, right-size clusters, optimize storage layouts, and manage job and warehouse costs.
- Use Terraform for Databricks and AWS infrastructure and promote changes across environments.
- Develop unit and integration tests, data quality checks, contract tests, replay procedures, and backfill processes.
- Maintain version control, code reviews, documentation, runbooks, monitoring, alerting, incident response, RCA, and SLO/SLA practices.
- Use authorized AI-assisted development tools for coding, refactoring, testing, troubleshooting, and documentation while validating outputs for correctness, performance, and security.
Requirements
- Formal training or certification in software engineering concepts and 5+ years of hands-on software development lifecycle experience.
- Experience building and maintaining data engineering pipelines.
- Hands-on experience building and operating a Databricks Lakehouse hosted in AWS.
- Experience with Delta Lake, including ACID tables, partitioning, and schema evolution.
- Proven experience with Spark on Databricks, including performance tuning, cluster sizing, skew mitigation, joins, caching, and file sizing.
- Experience with batch and streaming pipelines, Structured Streaming, incremental processing, backfills, and late-arriving data.
- Strong AWS data platform fundamentals, including S3, IAM, KMS, VPC, security groups, logging, and auditing.
- Experience implementing Databricks data governance and security controls such as Unity Catalog, table and column permissions, and credential passthrough patterns.
- Experience with monitoring, alerting, incident response, root-cause analysis, and SLO/SLA management.
- Understanding of responsible AI use in engineering workflows, data sensitivity, secure input and output handling, resiliency, and security expectations.
- Hands-on experience with enterprise-authorized AI-assisted software development tools and the ability to critically evaluate and refine generated outputs.
- Familiarity with modern front-end technologies is preferred.
- Exposure to cloud technologies is preferred.
Benefits
- Competitive total rewards package with base salary determined by role, experience, skill set, and location.
- Eligible roles may receive commission-based pay and/or discretionary incentive compensation in cash or forfeitable equity.
- Comprehensive health care coverage.
- On-site health and wellness centers.
- Retirement savings plan.
- Backup childcare.
- Tuition reimbursement.
- Mental health support and financial coaching.
Tech Stack
Categories
Data Engineering
About JPMorgan Chase
JPMorgan Chase provides consumer and commercial banking, payments, credit card, wealth management, and corporate and investment banking services to individuals, businesses, institutions, and governments. The public company (NYSE: JPM) earns revenue from interest, fees, trading, and asset management across operations in more than 100 markets. Headquartered in New York City with roots dating to 1799, it serves retail customers and prominent corporate and government clients through brands including Chase and J.P. Morgan.