Staff Software Engineer, AI Runtime
Databricks3 months ago
Base Salary
$190k - $265k/yr
Responsibilities
- Architect and evolve the managed GPU training platform for scalable, high-throughput, resilient training across thousands of accelerators.
- Solve large-scale training challenges involving multi-node orchestration, distributed parallelism, GPU scheduling, dynamic routing, data loading, and checkpoint and restore.
- Improve GPU utilization, training throughput, and cost efficiency across model architectures and hardware generations.
- Build resilience and observability foundations that detect and recover from hardware and software failures.
- Shape APIs, CLI, and developer experience for launching, monitoring, and debugging production training jobs.
- Lead engineering efforts from design through production rollout while maintaining high standards for performance, correctness, and reliability.
- Contribute to core AI Runtime systems and support new accelerators and regions.
- Mentor engineers, lead technical discussions, and help define Databricks’ long-term AI training infrastructure direction.
Requirements
- 10+ years of experience building and operating large-scale distributed systems, with significant depth in GPU training infrastructure, high-performance computing, or ML systems.
- Hands-on experience with PyTorch, FSDP, DeepSpeed, or Megatron and large-model parallelism strategies including data, tensor, pipeline, and sequence parallelism.
- Strong understanding of checkpointing, failure detection, and automatic recovery for long-running multi-node jobs.
- Understanding of GPU architecture, high-speed interconnects, collective communication, and training-performance bottlenecks, including NVLink and InfiniBand or RoCE.
- Experience building and operating managed, multi-tenant cloud platform products with availability, performance, and reliability SLAs and SLOs.
- Strong foundation in algorithms, data structures, and system design for performance-sensitive distributed systems.
- Proven ability to deliver technically complex initiatives with clear customer or business value.
- Strong communication and cross-functional collaboration skills.
- Strategic, product-oriented mindset and passion for mentoring engineers and fostering technical excellence.
- BS in Computer Science or a related field required; MS or PhD preferred.
Benefits
- Comprehensive employee benefits and perks, with details varying by region.
Tech Stack
Categories
About Databricks
Databricks is the Data and AI company. More than 20,000 organizations worldwide — including adidas, AT&T, Bayer, Block, Mastercard, Rivian, Unilever, and over 60% of the Fortune 500 — rely on Databricks to build and scale data and AI apps, analytics and agents. Headquartered in San Francisco with 30+ offices around the globe, Databricks offers a unified Data Intelligence Platform that includes Agent Bricks, Lakeflow, Lakehouse, Lakebase and Unity Catalog. --- Databricks applicants Please apply through our official Careers page at databricks.com/company/careers. All official communication from Databricks will come from email addresses ending with @databricks.com or @goodtime.io (our meeting tool).