LLM Inference Engineer (Mid, Sr, Staff)
Hippocratic AI1 month ago
Menlo Park, CA, USAMid Level / Senior / Staff+
H1B sponsor
Responsibilities
- Design and implement multi-node serving architectures for distributed LLM inference.
- Optimize multi-LoRA serving systems.
- Implement FP4/FP6 quantization techniques for transformer models.
- Implement speculative decoding, including draft-model and Eagle approaches.
- Develop disaggregated serving solutions with optimized caching for prefill and decoding.
- Benchmark and improve inference performance across deployment scenarios and GPU types.
- Establish performance optimization roadmaps and deploy measurable improvements to latency, throughput, and inference cost.
Requirements
- Experience optimizing LLM inference systems at scale.
- Proven expertise with distributed serving architectures for large language models.
- Hands-on experience implementing quantization techniques for transformer models.
- Strong understanding of speculative decoding and other modern inference optimization methods.
- Proficiency in Python and C++.
- Experience with CUDA programming and GPU optimization.
- Contributions to open-source inference frameworks such as vLLM, SGLang, or TensorRT-LLM are preferred.
- Experience with custom CUDA kernels, production inference deployments, and performance optimization systems is preferred.
Benefits
- Competitive compensation based on experience, expertise, and level of responsibility.
- Equity, health insurance, and other benefits.
- Menlo Park, California location with an expectation of working in the office five days per week.
Categories
About Hippocratic AI
Hippocratic AI builds a safety-focused large language model and AI agents for healthcare workflows, used by health systems for patient outreach, post-discharge follow-up, and chronic-care management. It licenses its platform and tools to providers to automate and scale clinical support tasks while meeting health-system requirements. Founded in 2023 and headquartered in Palo Alto, the company is privately held.