Staff Software Engineer, AI Runtime
Databricks5 months ago
Base Salary
$190k - $265k/yr
Responsibilities
- Architect and evolve the managed GPU training platform for scalable, high-throughput, resilient training across thousands of accelerators.
- Solve large-scale training challenges involving multi-node orchestration, distributed parallelism, GPU scheduling, dynamic routing, data loading, and checkpoint and restore.
- Improve GPU utilization, training throughput, and cost efficiency across model architectures and hardware generations.
- Build resilience and observability foundations that detect and recover from hardware and software failures.
- Shape APIs, CLI, and developer experience for launching, monitoring, and debugging production training jobs.
- Lead engineering efforts from design through production rollout while maintaining high standards for performance, correctness, and reliability.
- Contribute to core AI Runtime systems and support new accelerators and regions.
- Mentor engineers, lead technical discussions, and help define Databricks’ long-term AI training infrastructure direction.
Requirements
- 10+ years of experience building and operating large-scale distributed systems, with significant depth in GPU training infrastructure, high-performance computing, or ML systems.
- Hands-on experience with PyTorch, FSDP, DeepSpeed, or Megatron and large-model parallelism strategies including data, tensor, pipeline, and sequence parallelism.
- Strong understanding of checkpointing, failure detection, and automatic recovery for long-running multi-node jobs.
- Understanding of GPU architecture, high-speed interconnects, collective communication, and training-performance bottlenecks, including NVLink and InfiniBand or RoCE.
- Experience building and operating managed, multi-tenant cloud platform products with availability, performance, and reliability SLAs and SLOs.
- Strong foundation in algorithms, data structures, and system design for performance-sensitive distributed systems.
- Proven ability to deliver technically complex initiatives with clear customer or business value.
- Strong communication and cross-functional collaboration skills.
- Strategic, product-oriented mindset and passion for mentoring engineers and fostering technical excellence.
- BS in Computer Science or a related field required; MS or PhD preferred.
Benefits
- Comprehensive employee benefits and perks, with details varying by region.
Tech Stack
Categories
About Databricks
Databricks builds a cloud-based data and AI platform centered on the lakehouse architecture, combining data engineering, analytics, and machine learning with Apache Spark, Delta Lake, and MLflow. It sells subscriptions and cloud services to enterprises that need to unify data pipelines and develop large-scale AI and analytics. Founded in 2013 by the creators of Apache Spark and headquartered in San Francisco, the company is privately held and serves organizations across many industries.