Responsibilities
- Design and implement a distributed KV-cache system for transformer-based LLM intermediate states across GPUs and nodes.
- Optimize low-latency access, cache reuse, and eviction policies for long-context inputs, token streams, and embeddings.
- Integrate the cache with token streaming, batched decoding, model parallelism, and inference-serving pipelines.
- Develop consistency and synchronization protocols for multi-tenant and multi-request environments.
- Implement memory-aware sharding, eviction, replication, and GPU or distributed-memory strategies.
- Monitor performance and improve caching algorithms to reduce inference compute cost and response time.
- Evaluate open-source KV stores and build custom GPU-aware caching layers using technologies such as CUDA, Triton, shared memory, and RDMA.
Requirements
- PhD in Computer Science, Applied Mathematics, Electrical Engineering, or a related technical field.
- Strong understanding of transformer model internals and KV caching for autoregressive decoding.
- Experience with distributed systems, memory management, low-latency serving, RPC, gRPC, and CUDA-aware networking.
- Familiarity with NVIDIA GPUs, TensorRT, and Triton Inference Server.
- Proficiency in systems programming languages such as C++, Rust, Go, or CUDA.
- Preferred experience building LLM inference-serving systems with vLLM, SGLang, FasterTransformer, DeepSpeed, or Hugging Face Text Generation Inference.
- Preferred experience optimizing HBM, NUMA, and NVLink memory hierarchies and using NCCL, GDR, GDS, or InfiniBand.
- Preferred exposure to cache-aware scheduling, batching, and prefetching in model serving.
Benefits
- Opportunities to publish research, contribute to open source, and attend top conferences.
- Competitive compensation and generous research resources.
- Opportunity to work on AI systems supporting products used by millions and deployed at scale.
- Collaborative, innovation-driven research and engineering environment.
Categories
About ByteDance
ByteDance is a global incubator of platforms at the cutting edge of commerce, content, entertainment and enterprise services - over 2.5bn people interact with ByteDance products including TikTok. Creation is the core of ByteDance's purpose. Our products are built to help imaginations thrive. This is doubly true of the teams that make our innovations possible. Together, we inspire creativity and enrich life - a mission we aim towards achieving every day. At ByteDance, we create together and grow together. That's how we drive impact - for ourselves, our company, and the users we serve. We are committed to building a safe, healthy and positive online environment for all our users. We have over 110,000 employees based in more than 30 countries globally. Join us.