Lead Software Engineer - Pyspark, AWS, Python
JPMorgan Chase2 days ago
Pune, IndiaStaff+
Responsibilities
- Lead the design, development, maintenance, and optimization of scalable cloud-based data processing pipelines and infrastructure.
- Architect enterprise data models and large-scale data processing systems for efficient storage, retrieval, analytics, data quality, and accessibility.
- Translate business requirements into scalable data engineering solutions and align data practices with organizational objectives.
- Define data strategy and manage end-to-end data infrastructure from design and construction through installation and ongoing maintenance.
- Author, review, and approve technical requirements and architectural designs, and lead process re-engineering initiatives.
- Lead design workshops, coding sessions, hackathons, and adoption of reusable data engineering patterns.
- Drive data governance, regulatory compliance, data quality, and responsible handling of sensitive data.
- Lead effective use of approved AI-assisted development tools, including validation for correctness, performance, security, and compliance.
Requirements
- Formal training or certification in software engineering concepts and at least 5 years of applied experience.
- Expertise in Apache Spark or another distributed data processing framework, cloud data lake or lakehouse platforms such as AWS or Databricks, orchestration tools such as Airflow or AWS Step Functions, and relational and NoSQL databases.
- Proficiency in Python, SQL, and at least one additional programming language such as Java or Scala.
- Experience developing and optimizing Apache Spark pipelines for large-scale batch and real-time analytics, data transformation, and machine learning use cases.
- Knowledge of data structures, JSON, Avro, Protobuf, Parquet, Iceberg, data modeling techniques, microservices, serverless computing, Docker, and Kubernetes.
- Experience with test-driven or behavior-driven development, continuous integration and deployment tools, and streaming platforms such as Kafka or MQ.
- Experience leading AI-assisted software development practices and coaching engineers on secure, compliant, and responsible use of AI tools.
- Preferred experience with Terraform, AWS CloudFormation, Spinnaker, Snowflake, budgeting and resource allocation, and vendor relationship management.
Tech Stack
Apache AirflowApache HadoopApache KafkaApache SparkAWSDatabricksDockerJavaKubernetesPythonScalaSnowflakeSpinnakerSQLTerraform
Categories
Data Engineering
About JPMorgan Chase
JPMorgan Chase provides consumer and commercial banking, payments, credit card, wealth management, and corporate and investment banking services to individuals, businesses, institutions, and governments. The public company (NYSE: JPM) earns revenue from interest, fees, trading, and asset management across operations in more than 100 markets. Headquartered in New York City with roots dating to 1799, it serves retail customers and prominent corporate and government clients through brands including Chase and J.P. Morgan.