Principal Software Engineer - LLM Optimization
JPMorgan Chase6 days ago
Responsibilities
- Own systematic benchmarking and performance characterization for production LLM workloads, including reproducible baselines, regression detection, and configuration-impact measurement.
- Design and execute quantization experiments and evaluate accuracy, throughput, memory, and cost-per-token tradeoffs.
- Drive speculative decoding strategy, acceptance-rate measurement, and workload-specific configuration recommendations.
- Build and maintain GPU efficiency scorecards covering utilization, memory headroom, cost per 1K tokens, and waste.
- Benchmark the platform against external providers and published industry results to guide improvements.
- Lead evaluations of inference engine upgrades, scheduler architectures, async tensor parallelism, disaggregated prefill/decode, and advanced speculative decoding before production release.
- Collaborate on KV-cache optimization, prefix caching, and multi-node serving architecture.
- Design and run GPU chaos engineering scenarios, hardware diagnostics, and detection and recovery measurements.
- Architect and govern agentic AI-enabled engineering workflows, including AI-assisted code review, test generation, release readiness, incident triage, and root-cause analysis.
- Apply software development lifecycle tooling and enterprise-authorized AI automation to improve delivery and operational outcomes at scale.
Requirements
- Formal training or certification in software engineering concepts and at least 7 years of applied experience.
- Deep hands-on experience with production LLM inference systems such as vLLM, TensorRT-LLM, SGLang, LLM-D, or equivalent serving engines.
- Strong understanding of GPU memory architecture, KV-cache sizing, memory-bandwidth and compute bottlenecks, and inference-time quantization implications.
- Experience with quantization techniques and their production tradeoffs at scale.
- Familiarity with speculative decoding and the factors affecting production acceptance rates.
- Experience with rigorous benchmarking using GuideLLM, custom harnesses, or equivalent tools.
- Experience operating cloud GPU infrastructure at scale, including AWS, EKS, and managed inference services.
- Experience applying industry benchmarks to improve platforms and communicating technical tradeoffs to senior engineering and business stakeholders.
- Experience designing and leading adoption of agentic AI-enabled development practices across teams, including human-in-the-loop validation, auditability, traceability, and secure handling of sensitive data.
- Understanding of responsible AI use, engineering controls, security and resiliency implications, data sensitivity, and risk-based governance.
- Preferred experience with disaggregated prefill/decode serving, GPU hardware diagnostics, ML observability, and production monitoring.
Benefits
- Competitive total rewards package with base salary determined by role, experience, skills, and location; eligible roles may also receive commission-based pay or discretionary incentive compensation.
- Benefits may include comprehensive health care coverage, on-site health and wellness centers, retirement savings, backup childcare, tuition reimbursement, mental health support, and financial coaching.
- Employment offers are contingent on review of criminal conviction history under Section 19 of the Federal Deposit Insurance Act.
- JPMorganChase provides equal employment opportunity and reasonable accommodations for eligible needs.
Tech Stack
Categories
About JPMorgan Chase
With a history tracing its roots to 1799 in New York City, JPMorganChase is one of the world's oldest, largest, and best-known financial institutions—carrying forth the innovative spirit of our heritage firms in global operations across 100 markets. We serve millions of customers and many of the world’s most prominent corporate, institutional, and government clients daily, managing assets and investments, offering business advice and strategies, and providing innovative banking solutions and services. Social Media Terms and Conditions: https://bit.ly/JPMCSocialTerms JPMorgan Chase & Co. is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.