6 months ago
Base Salary
$195k - $365k/yr
Responsibilities
- Build and deploy high-throughput, ultra-low-latency inference engines for large language and foundational speech models.
- Optimize latency, throughput, time-to-first-token, and time-to-first-audio in real-time streaming environments.
- Implement continuous batching, KV cache management, stateful connections, and advanced decoding techniques.
- Optimize NVIDIA GPU memory hierarchy and eliminate hardware bottlenecks across GPU clusters.
- Develop real-time audio streaming, neural audio codec, and chunked audio generation systems.
- Deploy quantized models and distributed multi-GPU and multi-node inference pipelines.
- Manage autoscaling inference infrastructure using Kubernetes.
- Collaborate across the core ML training and backend infrastructure teams.
Requirements
- Hands-on experience building and deploying inference engines for large language models or foundational speech models.
- Deep understanding of continuous batching, KV cache management such as PagedAttention, and stateful real-time conversational AI connections.
- Strong knowledge of NVIDIA Ampere and Hopper GPU architectures and memory hierarchy.
- Experience with real-time inference performance tradeoffs involving latency, throughput, and time-to-first-token or time-to-first-audio.
- Preferred experience with vLLM, TensorRT-LLM, SGLang, or NVIDIA Triton Inference Server, including open-source contributions.
- Preferred experience with WebSockets or WebRTC, neural audio codecs, and chunked audio generation.
- Preferred experience with speculative decoding, lookahead decoding, or chunked prefill.
- Preferred experience with post-training quantization and FP8, INT8, AWQ, or GPTQ deployment.
- Preferred experience deploying multi-GPU and multi-node inference pipelines and managing Kubernetes autoscaling infrastructure.
- Clear communication, effective collaboration, and enthusiasm for systems engineering and speech AI.
Benefits
- $195K-$365K base salary plus performance bonus and equity.
- Top-tier medical, dental, and vision healthcare for employees and dependents with employer subsidy.
- 401(k) plan with company matching for full-time employees.
- Unlimited paid time off and 13 paid holidays.
- 12 weeks of paid parental leave regardless of gender.
- Hybrid schedule requiring at least three in-office days per week in San Francisco.
- Choice of top-of-the-line laptops or workstations, annual offsites, and a stocked office.
- Opportunity to join the early foundational team of Plaud’s SpeechLLM lab with meaningful ownership.
Tech Stack
Categories
About Plaud
Plaud builds AI-powered hardware devices and companion apps that capture, transcribe, and summarize meetings and other conversations for professionals. The company sells its consumer electronics and meeting software directly, with subscription services layered on its devices and cloud platform. Founded in 2021 and headquartered in San Francisco, Plaud is privately held and lists SOC 2, HIPAA, GDPR, and ISO 27001/27701 compliance.