Principal Software Engineer - LLM Optimization
JPMorgan Chase2 months ago
Base Salary
$204k - $285k/yr
Responsibilities
- Own systematic benchmarking and performance characterization for production LLM workloads, including reproducible baselines, regression detection, and configuration-impact measurement.
- Design and execute quantization experiments and evaluate accuracy, throughput, memory, and cost-per-token tradeoffs.
- Drive speculative decoding strategy, acceptance-rate measurement, and workload-specific configuration recommendations.
- Build and maintain GPU efficiency scorecards covering utilization, memory headroom, cost per 1K tokens, and waste.
- Benchmark the platform against external providers and published industry results to guide improvements.
- Lead evaluations of inference engine upgrades, scheduler architectures, async tensor parallelism, disaggregated prefill/decode, and advanced speculative decoding before production release.
- Collaborate on KV-cache optimization, prefix caching, and multi-node serving architecture.
- Design and run GPU chaos engineering scenarios, hardware diagnostics, and detection and recovery measurements.
- Architect and govern agentic AI-enabled engineering workflows, including AI-assisted code review, test generation, release readiness, incident triage, and root-cause analysis.
- Apply software development lifecycle tooling and enterprise-authorized AI automation to improve delivery and operational outcomes at scale.
Requirements
- Formal training or certification in software engineering concepts and at least 7 years of applied experience.
- Deep hands-on experience with production LLM inference systems such as vLLM, TensorRT-LLM, SGLang, LLM-D, or equivalent serving engines.
- Strong understanding of GPU memory architecture, KV-cache sizing, memory-bandwidth and compute bottlenecks, and inference-time quantization implications.
- Experience with quantization techniques and their production tradeoffs at scale.
- Familiarity with speculative decoding and the factors affecting production acceptance rates.
- Experience with rigorous benchmarking using GuideLLM, custom harnesses, or equivalent tools.
- Experience operating cloud GPU infrastructure at scale, including AWS, EKS, and managed inference services.
- Experience applying industry benchmarks to improve platforms and communicating technical tradeoffs to senior engineering and business stakeholders.
- Experience designing and leading adoption of agentic AI-enabled development practices across teams, including human-in-the-loop validation, auditability, traceability, and secure handling of sensitive data.
- Understanding of responsible AI use, engineering controls, security and resiliency implications, data sensitivity, and risk-based governance.
- Preferred experience with disaggregated prefill/decode serving, GPU hardware diagnostics, ML observability, and production monitoring.
Benefits
- Competitive total rewards package with base salary determined by role, experience, skills, and location; eligible roles may also receive commission-based pay or discretionary incentive compensation.
- Benefits may include comprehensive health care coverage, on-site health and wellness centers, retirement savings, backup childcare, tuition reimbursement, mental health support, and financial coaching.
- Employment offers are contingent on review of criminal conviction history under Section 19 of the Federal Deposit Insurance Act.
- JPMorganChase provides equal employment opportunity and reasonable accommodations for eligible needs.
Tech Stack
Categories
About JPMorgan Chase
JPMorgan Chase provides consumer and commercial banking, payments, credit card, wealth management, and corporate and investment banking services to individuals, businesses, institutions, and governments. The public company (NYSE: JPM) earns revenue from interest, fees, trading, and asset management across operations in more than 100 markets. Headquartered in New York City with roots dating to 1799, it serves retail customers and prominent corporate and government clients through brands including Chase and J.P. Morgan.