1 year ago
Responsibilities
- Design and build core systems that make large training runs fast and reliable
- Build scalable distributed training infrastructure for GPU clusters
- Implement and tune parallelism and sharding strategies for evolving architectures
- Optimize distributed efficiency through topology-aware collectives, communication and computation overlap, and straggler mitigation
- Build data loading systems that eliminate I/O bottlenecks for multimodal datasets
- Develop checkpointing mechanisms that balance memory constraints with recovery needs
- Create monitoring, profiling, and debugging tools for training stability and performance
Requirements
- Hands-on experience building distributed training infrastructure with PyTorch Distributed DDP/FSDP, DeepSpeed ZeRO, or Megatron-LM TP/PP
- Experience diagnosing performance bottlenecks and failure modes, including profiling, NCCL and collective-operation issues, hangs, out-of-memory errors, and stragglers
- Understanding of hardware accelerators and networking topologies
- Experience optimizing data pipelines for machine learning workloads
- MoE training experience is preferred
- Experience with large-scale distributed training across 100 or more GPUs is preferred
- Open-source contributions to training infrastructure projects are preferred
Benefits
- Competitive base salary with equity in a unicorn-stage company
- 100% of medical, dental, and vision premiums covered for employees and dependents
- 401(k) matching up to 4% of base pay
- Unlimited PTO and company-wide Refill Days
- San Francisco and Boston are preferred, but other locations are accepted
Tech Stack
Categories
About Liquid AI
Liquid AI builds general-purpose AI systems that run efficiently from data center accelerators to on-device hardware, emphasizing low latency, memory efficiency, privacy, and reliability. The company partners with enterprises in consumer electronics, automotive, life sciences, and financial services to deploy and benchmark models for real-world workloads. Founded in 2023 out of MIT CSAIL and headquartered in Cambridge, Massachusetts, it is privately held.