20 days ago
Bengaluru, IndiaSenior
Responsibilities
- Own production LLM inference services from model handoff through release engineering, capacity planning, cost optimization, and incident response.
- Tune inference performance to reduce end-to-end latency and improve throughput under real production traffic.
- Optimize inference runtimes and servers across heterogeneous GPU fleets, including schedulers, KV caches, batching, and memory systems.
- Build benchmarking suites, metrics, and tooling for latency, throughput, GPU utilization, memory, and cost.
- Improve monitoring, tracing, alerting, reliability, and incident-response practices, including postmortems.
- Evaluate and implement inference optimizations such as quantization, paging, and kernel or runtime improvements.
- Partner with data science and product teams to translate business needs into performance and availability SLOs.
Requirements
- 8+ years of strong software development experience.
- Experience deploying and operating LLM inference services in production.
- Strong production coding skills in Python and Go, including systems-level implementation and debugging.
- Experience with PyTorch, vLLM, SGLang, and/or TensorRT.
- Knowledge of GPU architecture and performance, including profiling and memory bandwidth/latency tradeoffs; CUDA or kernel programming is a strong plus.
- Understanding of LLM inference and optimization techniques such as continuous batching, KV-cache management, quantization, and optionally speculative decoding.
- Hands-on experience with performance optimization and systems programming for AI/ML workloads.
- Demonstrated ability to deliver measurable production improvements such as higher throughput, lower p95/p99 latency, or reduced GPU cost.
- Strong root-cause analysis skills across models, runtimes, networking, and infrastructure.
- Proficiency applying autonomous AI coding agents, advanced prompting, and human-in-the-loop code review to improve software delivery speed and accuracy.
About eBay
eBay operates a global online marketplace where individuals and businesses buy and sell goods via auctions and fixed‑price listings. It generates revenue from transaction fees, advertising, and payment processing on its managed marketplace. Founded in 1995 and headquartered in San Jose, it is a publicly traded company on NASDAQ and serves buyers and sellers in over 190 markets.