Lead Software Engineer - Databricks/PySpark/AI
JPMorgan Chase4 days ago
Responsibilities
- Build and optimize data pipelines and workflows that provide reliable access to structured and unstructured data for agentic AI systems.
- Develop data retrieval, indexing, APIs, data services, and function endpoints that allow AI agents to search enterprise data, invoke tools, and execute tasks.
- Build orchestration and state-management layers supporting agent memory, context persistence, and multi-step task chaining.
- Write, review, and deploy secure, production-quality code using AWS best practices, automated testing, and CI/CD pipelines.
- Establish standards for validating AI-assisted development outputs, including correctness, performance, security, data sensitivity, and compliance.
- Implement data management, governance, data quality, observability, resiliency, and security practices for data consumed and generated by AI agents.
- Mentor junior and mid-level engineers through pairing, code reviews, and technical coaching.
- Collaborate with product owners, data scientists, and business stakeholders to deliver production-ready agentic AI solutions.
- Evaluate and adopt agentic AI frameworks, tool-use patterns, and data engineering practices.
Requirements
- Formal training or certification in software engineering concepts and 5+ years of applied experience.
- Expert-level Python and PySpark programming skills with a strong portfolio of production-grade code.
- Extensive hands-on experience with Databricks and the AWS cloud ecosystem, including AWS Glue, S3, SQS/SNS, and Lambda.
- Strong experience with Lakehouse/Delta Lake architecture, application development, testing, operational stability, Snowflake, Terraform, LLMs, data observability, data quality, query optimization, and cost optimization.
- In-depth knowledge of enterprise-scale big data and data warehousing concepts.
- Extensive experience with CI/CD processes and automated testing frameworks.
- Understanding of agile and DevOps practices, application resiliency, security measures, and responsible AI use in engineering workflows.
- Experience building APIs, data services, and retrieval systems connecting AI agents with enterprise data.
- Preferred experience with LangGraph, AutoGen, CrewAI, OpenAI Assistants API, vector databases such as Pinecone, FAISS, and Chroma, embedding workflows, agent memory, guardrails, safety frameworks, and agentic-system observability.
- Understanding of tool-use and function-calling patterns, retrieval-augmented generation, data provenance, auditability, and human-in-the-loop controls.
Benefits
- Competitive total rewards package with benefits including comprehensive health care coverage, on-site health and wellness centers, retirement savings plan, backup childcare, tuition reimbursement, mental health support, and financial coaching.
- Eligible roles may receive commission-based pay and/or discretionary incentive compensation in cash and/or forfeitable equity.
- Additional compensation and benefits details are provided during the hiring process.
Tech Stack
Categories
AI ApplicationsData Engineering
About JPMorgan Chase
JPMorgan Chase provides consumer and commercial banking, payments, credit card, wealth management, and corporate and investment banking services to individuals, businesses, institutions, and governments. The public company (NYSE: JPM) earns revenue from interest, fees, trading, and asset management across operations in more than 100 markets. Headquartered in New York City with roots dating to 1799, it serves retail customers and prominent corporate and government clients through brands including Chase and J.P. Morgan.