Principal Software Engineer - AI Foundations
JPMorgan Chase2 months ago
Base Salary
$204k - $285k/yr
Responsibilities
- Design, build, and operate a high-throughput, low-latency LLM serving platform with batching, scheduling, caching, streaming, multi-tenancy, and autoscaling across GPU and CPU fleets.
- Build and evolve a GenAI Gateway and inference API layer supporting authentication, authorization, quotas, rate limiting, routing, request shaping, policy enforcement, and observability.
- Develop intelligent model-routing systems across open-source and vendor model backends while balancing quality, latency, reliability, and cost.
- Optimize GPU serving through kernel-level tuning, model compilation and acceleration, memory management, KV-cache strategies, quantization, compression, and prefill/decode optimization.
- Design disaggregated serving and distributed inference architectures, including prefill/decode separation, KV-cache offload, and tiered serving.
- Create secure production code, review and improve peer code, and build reusable frameworks and platform components for multiple teams.
- Own SDK and service integrations and ensure their reliability, performance, and maintainability.
- Establish inference SLOs and SLAs and lead load testing, capacity planning, incident response, regression detection, benchmarking, observability, and automated remediation.
- Architect and govern agentic AI-enabled engineering workflows with guardrails for validation, security, resiliency, auditability, and reuse.
- Influence senior technical, business, and product stakeholders and align platform roadmaps with measurable business outcomes.
Requirements
- Formal software engineering training or certification plus 7+ years of applied experience delivering system design, application development, testing, and operational stability for large-scale platforms and services.
- Expert proficiency in one or more programming languages, such as Python, Java, Scala, or Go, with strong code quality, testing, and debugging practices.
- Experience designing and leading adoption of agentic AI-enabled development practices, including human-in-the-loop validation, auditability, traceability, and secure sensitive-data handling.
- Strong understanding of responsible AI, security, resiliency, data sensitivity, and risk-based governance in engineering workflows.
- Experience designing and operating high-scale inference and distributed systems with multi-tenancy, backpressure, load shedding, rate limiting, request prioritization, and tail-latency reduction.
- Strong GPU-serving knowledge and experience improving production efficiency and utilization across compute, memory, batching, concurrency, networking, and profiling concerns.
- Experience with model-serving stacks, registries and artifacts, canary and A/B rollouts, shadow traffic, and performance benchmarking.
- Practical cloud-native experience with containers, orchestration, infrastructure as code, observability, and production systems with defined SLOs.
- Experience solving complex technology problems across platform engineering, ML systems, data engineering, or distributed systems.
- Strong communication skills and the ability to influence senior leaders and executives by translating complex technical issues into clear decisions and trade-offs.
- Preferred experience with speculative decoding, KV-cache management, paged-attention approaches, optimized sampling, continuous batching, quantization and compression toolchains, disaggregated serving, model routing and governance, performance engineering, multi-tenant platforms, developer self-service, and security-by-design for ML/LLM systems.
Benefits
- Competitive total rewards package with base salary determined by role, experience, skill set, and location.
- Eligible roles may receive commission-based pay or discretionary incentive compensation in cash or forfeitable equity.
- Comprehensive health care coverage and on-site health and wellness centers.
- Retirement savings plan and backup childcare.
- Tuition reimbursement, mental health support, and financial coaching.
- Additional benefits and total compensation details are provided during the hiring process.
Categories
About JPMorgan Chase
JPMorgan Chase provides consumer and commercial banking, payments, credit card, wealth management, and corporate and investment banking services to individuals, businesses, institutions, and governments. The public company (NYSE: JPM) earns revenue from interest, fees, trading, and asset management across operations in more than 100 markets. Headquartered in New York City with roots dating to 1799, it serves retail customers and prominent corporate and government clients through brands including Chase and J.P. Morgan.