Senior Software Engineer, AI Runtime
Databricks5 months ago
Base Salary
$160k - $225k/yr
Responsibilities
- Drive the architecture and evolution of the managed GPU training platform across large accelerator fleets.
- Build systems for multi-node orchestration, distributed parallelism, GPU scheduling, dynamic routing, high-throughput data loading, checkpointing, and restore.
- Improve GPU utilization, training throughput, and cost efficiency across model architectures and hardware generations.
- Build resilience and observability foundations for detecting and recovering from hardware and software failures.
- Shape APIs, CLI, and developer experience for launching, monitoring, and debugging production training jobs.
- Lead engineering efforts from design through production rollout while maintaining high standards for performance, correctness, and reliability.
- Support new accelerators and regions as the platform fleet expands.
- Mentor engineers and contribute to Databricks’ technical direction in AI training infrastructure.
Requirements
- 5+ years of experience building and operating large-scale distributed systems, with experience in GPU training infrastructure, high-performance computing, or ML systems.
- Experience with distributed training frameworks such as PyTorch, FSDP, DeepSpeed, or Megatron, and with data, tensor, pipeline, and sequence parallelism.
- Strong understanding of checkpointing, failure detection, and automatic recovery for long-running, multi-node jobs.
- Knowledge of GPU architecture, high-speed interconnects, collective communication, and training throughput and utilization bottlenecks.
- Experience building and operating managed, multi-tenant cloud platform products with availability, performance, and reliability SLAs and SLOs.
- Strong foundation in algorithms, data structures, and system design for performance-sensitive distributed systems.
- Proven ability to deliver technically complex, high-impact initiatives with customer or business value.
- Strong communication and cross-functional collaboration skills, plus a customer-focused mindset and passion for mentoring engineers.
- BS in Computer Science or a related field; MS or PhD preferred.
Tech Stack
Categories
About Databricks
Databricks builds a cloud-based data and AI platform centered on the lakehouse architecture, combining data engineering, analytics, and machine learning with Apache Spark, Delta Lake, and MLflow. It sells subscriptions and cloud services to enterprises that need to unify data pipelines and develop large-scale AI and analytics. Founded in 2013 by the creators of Apache Spark and headquartered in San Francisco, the company is privately held and serves organizations across many industries.