Principal Software Engineer - AI Foundations
JPMorgan Chase13 hours ago
Responsibilities
- Design, build, and operate a high-throughput, low-latency LLM serving platform with batching, scheduling, caching, streaming, multi-tenancy, and autoscaling across GPU and CPU fleets.
- Build and evolve a GenAI Gateway and inference API layer supporting authentication, authorization, quotas, rate limiting, routing, request shaping, policy enforcement, and observability.
- Develop intelligent model-routing systems across open-source and vendor model backends while balancing quality, latency, reliability, and cost.
- Optimize GPU serving through kernel-level tuning, model compilation and acceleration, memory management, KV-cache strategies, quantization, compression, and prefill/decode optimization.
- Design disaggregated serving and distributed inference architectures, including prefill/decode separation, KV-cache offload, and tiered serving.
- Create secure production code, review and improve peer code, and build reusable frameworks and platform components for multiple teams.
- Own SDK and service integrations and ensure their reliability, performance, and maintainability.
- Establish inference SLOs and SLAs and lead load testing, capacity planning, incident response, regression detection, benchmarking, observability, and automated remediation.
- Architect and govern agentic AI-enabled engineering workflows with guardrails for validation, security, resiliency, auditability, and reuse.
- Influence senior technical, business, and product stakeholders and align platform roadmaps with measurable business outcomes.
Requirements
- Formal software engineering training or certification plus 7+ years of applied experience delivering system design, application development, testing, and operational stability for large-scale platforms and services.
- Expert proficiency in one or more programming languages, such as Python, Java, Scala, or Go, with strong code quality, testing, and debugging practices.
- Experience designing and leading adoption of agentic AI-enabled development practices, including human-in-the-loop validation, auditability, traceability, and secure sensitive-data handling.
- Strong understanding of responsible AI, security, resiliency, data sensitivity, and risk-based governance in engineering workflows.
- Experience designing and operating high-scale inference and distributed systems with multi-tenancy, backpressure, load shedding, rate limiting, request prioritization, and tail-latency reduction.
- Strong GPU-serving knowledge and experience improving production efficiency and utilization across compute, memory, batching, concurrency, networking, and profiling concerns.
- Experience with model-serving stacks, registries and artifacts, canary and A/B rollouts, shadow traffic, and performance benchmarking.
- Practical cloud-native experience with containers, orchestration, infrastructure as code, observability, and production systems with defined SLOs.
- Experience solving complex technology problems across platform engineering, ML systems, data engineering, or distributed systems.
- Strong communication skills and the ability to influence senior leaders and executives by translating complex technical issues into clear decisions and trade-offs.
- Preferred experience with speculative decoding, KV-cache management, paged-attention approaches, optimized sampling, continuous batching, quantization and compression toolchains, disaggregated serving, model routing and governance, performance engineering, multi-tenant platforms, developer self-service, and security-by-design for ML/LLM systems.
Benefits
- Competitive total rewards package with base salary determined by role, experience, skill set, and location.
- Eligible roles may receive commission-based pay or discretionary incentive compensation in cash or forfeitable equity.
- Comprehensive health care coverage and on-site health and wellness centers.
- Retirement savings plan and backup childcare.
- Tuition reimbursement, mental health support, and financial coaching.
- Additional benefits and total compensation details are provided during the hiring process.
Categories
About JPMorgan Chase
With a history tracing its roots to 1799 in New York City, JPMorganChase is one of the world's oldest, largest, and best-known financial institutions—carrying forth the innovative spirit of our heritage firms in global operations across 100 markets. We serve millions of customers and many of the world’s most prominent corporate, institutional, and government clients daily, managing assets and investments, offering business advice and strategies, and providing innovative banking solutions and services. Social Media Terms and Conditions: https://bit.ly/JPMCSocialTerms JPMorgan Chase & Co. is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.