LLM Inference Systems Engineer
Hippocratic AI1 month ago
Responsibilities
- Design, implement, and operate disaggregated LLM serving architectures with independent prefill and decode pools, KV-cache transfer, routing, batching, and failure recovery.
- Optimize multi-node inference across GPU systems and high-performance communication paths.
- Build and tune serving systems using LLM inference frameworks such as SGLang, vLLM, NVIDIA Dynamo, and TensorRT-LLM.
- Profile and improve compute, memory, network, and storage performance, including time to first token, inter-token latency, throughput, tail latency, and cost per token.
- Develop benchmarking and capacity-planning workflows across model architectures, GPU types, parallelism strategies, and concurrency levels.
- Productionize serving workloads on Kubernetes with health checks, autoscaling, safe rollouts, metrics, tracing, and diagnostics.
- Diagnose distributed failures including collective timeouts, topology mismatches, packet loss, congestion, KV-transfer stalls, GPU out-of-memory errors, and uneven load.
- Partner with model researchers and platform engineers to launch models, serving features, and hardware generations safely.
Requirements
- Production experience building or operating distributed systems, high-performance computing systems, or large-scale ML inference platforms.
- Strong understanding of LLM inference, including tensor, pipeline, and data parallelism, continuous batching, KV-cache management, and prefill/decode behavior.
- Hands-on experience with GPU communication and networking technologies such as NCCL, NVLink/NVSwitch, InfiniBand, RoCE, RDMA, and UCX.
- Strong Python skills and working proficiency in C++ or another systems language.
- Ability to profile and debug performance across application, runtime, kernel, network, and infrastructure layers.
- Experience deploying and operating production workloads on Kubernetes with reliability and observability standards.
- Clear written and verbal communication across research, infrastructure, and product-facing teams.
- Master's degree or PhD required.
- Preferred experience with disaggregated serving, KV-cache transfer, NVIDIA Dynamo/NIXL, SGLang, vLLM, TensorRT-LLM, NVLS, SHARP, GPUDirect RDMA, UCX, RDMA congestion control, GPU-cluster topology optimization, CUDA, Triton, custom kernels, speculative decoding, multi-LoRA serving, quantization, cache-aware routing, open-source distributed-systems or inference projects, and benchmarking new GPU platforms.
Benefits
- Full-time role expected to be performed in the Menlo Park office five days per week unless otherwise specified.
- Opportunity to work on a safety-focused healthcare LLM platform with experts from healthcare, AI, research, and technology organizations.
Tech Stack
Categories
About Hippocratic AI
Hippocratic AI builds a safety-focused large language model and AI agents for healthcare workflows, used by health systems for patient outreach, post-discharge follow-up, and chronic-care management. It licenses its platform and tools to providers to automate and scale clinical support tasks while meeting health-system requirements. Founded in 2023 and headquartered in Palo Alto, the company is privately held.