10 hours ago
Bengaluru, IndiaMid Level
Responsibilities
- Own production LLM inference services from model handoff through release engineering, capacity planning, cost optimization, and incident response.
- Tune inference performance to reduce end-to-end latency and increase throughput across real production traffic.
- Scale inference across heterogeneous GPU fleets and optimize vLLM, Triton, schedulers, KV cache, batching, and memory components.
- Build benchmarking suites, metrics, and tooling for latency, throughput, GPU utilization, memory, and cost.
- Improve monitoring, tracing, and alerting and participate in incident response and postmortems.
- Evaluate and implement inference optimizations such as quantization, paging, and kernel or runtime improvements.
- Partner with data science and product teams to translate business requirements into performance and availability SLOs.
- Apply autonomous AI coding agents, advanced prompting, and human-in-the-loop code review to improve software delivery speed and accuracy.
Requirements
- At least 5 years of strong development experience.
- Experience deploying and operating LLM inference services in production.
- Strong production coding skills in Python plus Go or Rust, including systems-level implementation and debugging.
- Experience with PyTorch, vLLM, SGLang, and/or TensorRT.
- Knowledge of GPU architecture and performance, including profiling and memory bandwidth and latency tradeoffs; CUDA or kernel programming is a strong plus.
- Understanding of LLM inference optimization techniques such as continuous batching, KV cache management, quantization, and speculative decoding.
- At least 3 years of hands-on experience in performance optimization and systems programming for AI/ML workloads.
- Demonstrated ability to deliver measurable production improvements such as higher throughput, lower p95/p99 latency, or reduced GPU cost.
- Root-cause analysis skills across models, runtimes, networking, and infrastructure.
- Proficiency applying autonomous AI coding agents, advanced prompting, and careful human-in-the-loop code review.
Categories
About TCGplayer
TCGplayer operates an online marketplace and software tools for buying and selling trading card games and related collectibles, serving hobby shops and individual sellers. It generates revenue through marketplace fees, seller subscriptions, and fulfillment programs such as TCGplayer Direct. An eBay subsidiary, the company provides authentication and logistics services that help stores list inventory at scale and reach buyers across the U.S. and internationally.