Responsibilities
- Own the stability, performance optimization, and issue resolution of large-scale generative AI training and inference systems.
- Design and improve resource scheduling systems, including utilization, prioritization, fine-grained scheduling, and scheduling explainability.
- Plan, deploy, operate, and optimize high-speed RDMA networks and heterogeneous GPU, NPU, and XPU clusters.
- Own the architecture, selection, operation, performance, reliability, and stability of high-performance storage systems.
- Build capacity planning methods and deliver reliable, timely, and cost-effective resource expansion and configuration optimization.
- Develop unified compute pools for co-located training and inference workloads and establish operating methods for compute efficiency and cost control.
- Build monitoring, alerting, and automated remediation across the training and inference path and resolve complex production incidents.
- Partner with algorithm, product, and operations teams on MLOps solutions and establish engineering practices for large-scale generative AI systems.
Requirements
- Bachelor's degree or above in Computer Science, Software Engineering, or a related field.
- 3+ years of experience in large-scale machine learning, generative AI MLOps, or closely related infrastructure work.
- Proficiency in Python and/or Go, strong Linux development and operations skills, and expert shell scripting and automation capabilities.
- Expertise with Docker or Kata containerization and Kubernetes orchestration, including hands-on operation and optimization of large-scale heterogeneous GPU or NPU clusters.
- Knowledge and practical optimization experience with RDMA high-speed networking and distributed or object storage.
- Expertise designing and optimizing resource scheduling systems and improving scheduling policies and utilization.
- Experience maintaining and optimizing large-scale generative AI training or inference systems and resolving issues across the full training and inference path.
- Experience with co-located training and inference compute pools or offline inference efficiency, plus strong communication, collaboration, ownership, and production operations skills.
- Preferred qualifications include productionizing large-scale generative AI models; managing heterogeneous clusters and capacity planning in multi-cloud or hybrid-cloud environments; tuning distributed training and inference systems; optimizing RDMA or storage performance; delivering compute capacity and operating frameworks with measurable cost or utilization improvements; designing distributed systems; and running fault-injection or chaos exercises.
About ByteDance
ByteDance is a global incubator of platforms at the cutting edge of commerce, content, entertainment and enterprise services - over 2.5bn people interact with ByteDance products including TikTok. Creation is the core of ByteDance's purpose. Our products are built to help imaginations thrive. This is doubly true of the teams that make our innovations possible. Together, we inspire creativity and enrich life - a mission we aim towards achieving every day. At ByteDance, we create together and grow together. That's how we drive impact - for ourselves, our company, and the users we serve. We are committed to building a safe, healthy and positive online environment for all our users. We have over 110,000 employees based in more than 30 countries globally. Join us.