10 hours ago
Bengaluru, IndiaMid Level
Responsibilities
- Own production LLM inference from model handoff through serving, release engineering, capacity planning, cost optimization, and incident response.
- Optimize inference performance across production traffic by reducing latency and increasing throughput.
- Scale and optimize inference runtimes and servers across heterogeneous GPU fleets, including vLLM, Triton, schedulers, KV cache, batching, and memory systems.
- Build benchmarking suites, metrics, and tooling for latency, throughput, GPU utilization, memory, and cost.
- Improve monitoring, tracing, alerting, incident response, and postmortem-driven reliability.
- Evaluate and implement inference optimizations such as quantization, paging, and kernel or runtime improvements.
- Partner with data science and product teams to translate business requirements into performance and availability SLOs.
Requirements
- 5+ years of strong development experience.
- Experience deploying and operating LLM inference services in production.
- Strong Python skills plus Go or Rust for systems-level implementation and debugging.
- Experience with PyTorch and ML inference frameworks or runtimes including vLLM, SGLang, and/or TensorRT.
- Knowledge of GPU architecture and performance, including profiling and memory bandwidth/latency tradeoffs; CUDA or kernel programming is a plus.
- 3+ years of hands-on experience in performance optimization and systems programming for AI/ML workloads.
- Understanding of LLM inference techniques including continuous batching, KV cache management, quantization, and preferably speculative decoding.
- Demonstrated ability to deliver measurable production improvements such as higher throughput, lower p95/p99 latency, or reduced GPU cost.
- Skill in root-cause analysis across models, runtimes, networking, and infrastructure.
- Proficiency applying autonomous AI coding agents, advanced prompting, and human-in-the-loop code review to improve software delivery speed and accuracy.
About TCGplayer
TCGplayer operates an online marketplace and software tools for buying and selling trading card games and related collectibles, serving hobby shops and individual sellers. It generates revenue through marketplace fees, seller subscriptions, and fulfillment programs such as TCGplayer Direct. An eBay subsidiary, the company provides authentication and logistics services that help stores list inventory at scale and reach buyers across the U.S. and internationally.