3 months ago
Base Salary
$230k - $300k/yr
Responsibilities
- Lead the technical direction, architecture decisions, and engineering execution for the model serving platform.
- Build execution runtimes, batching, scheduling, and distributed inference components.
- Develop high-performance C++ and CUDA/HIP modules, custom GPU kernels, and memory-optimized runtimes.
- Collaborate with ML researchers to productionize multimodal models and deliver low-latency, scalable inference.
- Build Python APIs and services for downstream real-time AI applications.
- Mentor engineers through code reviews, design discussions, and hands-on technical guidance.
- Drive profiling, benchmarking, observability, testing, monitoring, reliability, and performance optimization.
- Troubleshoot complex issues across GPU, runtime, and service layers.
Requirements
- Bachelor’s degree in Computer Science, Computer Engineering, Electrical Engineering, or equivalent practical experience.
- At least 5 years of experience designing and building scalable, reliable backend systems or distributed infrastructure.
- Strong understanding of LLM inference mechanics, including prefill, decode, batching, and KV cache.
- Experience with Kubernetes, Ray, and containerization.
- Strong proficiency in C++ and Python.
- Strong system-level debugging, profiling, and performance optimization skills.
- Ability to collaborate with ML researchers and translate model or runtime requirements into production systems.
- Ability to lead technical discussions, mentor engineers, and drive engineering quality.
- Preferred experience with ML systems engineering, distributed GPU scheduling, vLLM, Sglang, TRT-LLM, large-scale ML/MLOps infrastructure, CUDA or ROCm, GPU profiling tools, multimodal model architectures, efficient inference, or open-source ML/HPC infrastructure contributions.
Benefits
- Medical, dental, and vision insurance.
- 401k plan.
- Daily lunch, snacks, and beverages.
- Flexible time off.
- Competitive salary and equity.
- The role is based in the office, with employees expected to be comfortable working from the office.
Tech Stack
Categories
