Responsibilities
- Own the stability, performance, and optimization of large-scale training and heterogeneous inference systems for generative AI workloads.
- Design and improve flexible resource scheduling systems, including scheduling efficiency, utilization, explainability, fine-grained scheduling, and workload prioritization.
- Plan, deploy, operate, and optimize high-speed RDMA networks and heterogeneous GPU, NPU, and XPU compute clusters.
- Own the architecture, selection, operation, reliability, and performance of high-performance distributed and object storage systems.
- Develop capacity planning methodology, forecast growth, and deliver reliable and cost-effective resource expansion and configuration optimization.
- Build unified compute pools for co-located training and inference workloads and establish operating practices for compute efficiency and cost control.
- Build monitoring, alerting, and automated remediation across training and inference systems and resolve complex production incidents.
- Partner with algorithm, product, and operations teams to deliver MLOps solutions and establish engineering practices for large-scale generative AI.
Requirements
- Bachelor's degree or above in Computer Science, Software Engineering, or a related field.
- 3+ years of experience in large-scale machine learning, generative AI MLOps, or closely related infrastructure work.
- Proficiency in Python and/or Go, solid Linux development and operations skills, and expert shell scripting and automation-tooling abilities.
- Expertise with Docker or Kata containerization and Kubernetes orchestration, including hands-on operation and optimization of large-scale heterogeneous compute clusters.
- Knowledge and practical optimization experience with RDMA high-speed networking and distributed or object storage.
- Expertise designing and optimizing resource scheduling systems, including scheduling policies, utilization, and scheduling explainability.
- Experience stabilizing and optimizing large-scale generative AI training or inference systems and resolving issues across the full training and inference path.
- Project experience with co-located training and inference compute pools or offline inference efficiency.
- Strong communication, cross-team collaboration, self-direction, ownership, and ability to work under large-scale production pressure.
- Preferred: end-to-end experience productionizing large-scale generative AI models such as LLM, text-to-image, or text-to-video models.
- Preferred: experience managing heterogeneous clusters and capacity planning in multi-cloud or hybrid-cloud environments.
- Preferred: familiarity with distributed training frameworks, inference optimization tooling, network performance tuning, storage I/O optimization, large-scale capacity delivery, distributed system architecture, and fault-injection or chaos exercises.
Tech Stack
Categories
About ByteDance
ByteDance is a global incubator of platforms at the cutting edge of commerce, content, entertainment and enterprise services - over 2.5bn people interact with ByteDance products including TikTok. Creation is the core of ByteDance's purpose. Our products are built to help imaginations thrive. This is doubly true of the teams that make our innovations possible. Together, we inspire creativity and enrich life - a mission we aim towards achieving every day. At ByteDance, we create together and grow together. That's how we drive impact - for ourselves, our company, and the users we serve. We are committed to building a safe, healthy and positive online environment for all our users. We have over 110,000 employees based in more than 30 countries globally. Join us.