Responsibilities
- Design and implement distributed KV-cache systems for storing and retrieving transformer attention key/value states across GPUs and nodes.
- Optimize low-latency cache access, eviction policies, memory-aware sharding, and replication for long-context inputs, token streams, and reused embeddings.
- Integrate caching with token streaming, batched decoding, model parallelism, and inference-serving systems.
- Develop cache consistency and synchronization protocols for multi-tenant and multi-request environments.
- Monitor performance and improve caching algorithms to reduce inference compute costs and response times.
- Evaluate and extend open-source KV stores or build custom GPU-aware caching layers using technologies such as CUDA, Triton, shared memory, and RDMA.
Requirements
- PhD in Computer Science, Applied Mathematics, Electrical Engineering, or a related technical field.
- Deep understanding of transformer-based model internals and the effect of KV caching on autoregressive decoding.
- Experience with distributed systems, memory management, and low-latency serving, including RPC, gRPC, and CUDA-aware networking.
- Familiarity with high-performance computing environments, NVIDIA GPUs, TensorRT, and Triton Inference Server.
- Proficiency in systems-level development using one or more of C++, Rust, Go, or CUDA.
- Preferred experience building LLM inference-serving systems with vLLM, SGLang, FasterTransformer, DeepSpeed, or Hugging Face Text Generation Inference.
- Preferred experience with HBM, NUMA, NVLink, NCCL, GDR, GDS, and InfiniBand.
- Preferred exposure to cache-aware scheduling, batching, and prefetching strategies for model serving.
Benefits
- Opportunities to publish research, contribute to open source, attend top conferences, and access generous research resources.
- Competitive compensation and an innovation-driven culture.
Categories
About ByteDance
ByteDance is a global incubator of platforms at the cutting edge of commerce, content, entertainment and enterprise services - over 2.5bn people interact with ByteDance products including TikTok. Creation is the core of ByteDance's purpose. Our products are built to help imaginations thrive. This is doubly true of the teams that make our innovations possible. Together, we inspire creativity and enrich life - a mission we aim towards achieving every day. At ByteDance, we create together and grow together. That's how we drive impact - for ourselves, our company, and the users we serve. We are committed to building a safe, healthy and positive online environment for all our users. We have over 110,000 employees based in more than 30 countries globally. Join us.