10 hours ago
Bengaluru, IndiaMid Level
Responsibilities
- Own production LLM inference services from model handoff through serving, release engineering, capacity planning, cost optimization, and incident response.
- Tune inference performance to reduce end-to-end latency and increase throughput across real production traffic.
- Scale and optimize inference runtimes and servers across heterogeneous GPU fleets, including schedulers, KV caches, batching, and memory management.
- Build benchmarking suites, metrics, and tooling for latency, throughput, GPU utilization, memory, and cost.
- Improve monitoring, tracing, alerting, incident response, and postmortem practices to strengthen reliability.
- Evaluate and implement inference optimizations such as quantization, paging, and kernel or runtime improvements.
- Partner with data science and product teams to translate business requirements into performance and availability SLOs.
Requirements
- At least 5 years of strong development experience.
- Experience deploying and operating LLM inference services in production.
- Strong production coding skills in Python plus Go or Rust, including systems-level implementation and debugging.
- Experience with ML frameworks and runtimes such as PyTorch, vLLM, SGLang, and/or TensorRT.
- Knowledge of GPU architecture and performance profiling, including memory bandwidth and latency tradeoffs; CUDA or kernel programming is a plus.
- Understanding of LLM inference optimization techniques including continuous batching, KV-cache management, quantization, and optionally speculative decoding.
- At least 3 years of hands-on performance optimization and systems programming for AI/ML workloads.
- Demonstrated ability to deliver measurable production improvements such as higher throughput, lower p95/p99 latency, or reduced GPU cost.
- Root-cause analysis skills spanning models, runtimes, networking, and infrastructure.
- Experience applying autonomous AI coding agents, advanced prompting, and human-in-the-loop code review to improve software delivery speed and code accuracy.
About TCGplayer
TCGplayer operates an online marketplace and software tools for buying and selling trading card games and related collectibles, serving hobby shops and individual sellers. It generates revenue through marketplace fees, seller subscriptions, and fulfillment programs such as TCGplayer Direct. An eBay subsidiary, the company provides authentication and logistics services that help stores list inventory at scale and reach buyers across the U.S. and internationally.