Senior Software Engineer, AI Runtime
Databricks3 months ago
Base Salary
$160k - $225k/yr
Responsibilities
- Drive the architecture and evolution of the managed GPU training platform across large accelerator fleets.
- Build systems for multi-node orchestration, distributed parallelism, GPU scheduling, dynamic routing, high-throughput data loading, checkpointing, and restore.
- Improve GPU utilization, training throughput, and cost efficiency across model architectures and hardware generations.
- Build resilience and observability foundations for detecting and recovering from hardware and software failures.
- Shape APIs, CLI, and developer experience for launching, monitoring, and debugging production training jobs.
- Lead engineering efforts from design through production rollout while maintaining high standards for performance, correctness, and reliability.
- Support new accelerators and regions as the platform fleet expands.
- Mentor engineers and contribute to Databricks’ technical direction in AI training infrastructure.
Requirements
- 5+ years of experience building and operating large-scale distributed systems, with experience in GPU training infrastructure, high-performance computing, or ML systems.
- Experience with distributed training frameworks such as PyTorch, FSDP, DeepSpeed, or Megatron, and with data, tensor, pipeline, and sequence parallelism.
- Strong understanding of checkpointing, failure detection, and automatic recovery for long-running, multi-node jobs.
- Knowledge of GPU architecture, high-speed interconnects, collective communication, and training throughput and utilization bottlenecks.
- Experience building and operating managed, multi-tenant cloud platform products with availability, performance, and reliability SLAs and SLOs.
- Strong foundation in algorithms, data structures, and system design for performance-sensitive distributed systems.
- Proven ability to deliver technically complex, high-impact initiatives with customer or business value.
- Strong communication and cross-functional collaboration skills, plus a customer-focused mindset and passion for mentoring engineers.
- BS in Computer Science or a related field; MS or PhD preferred.
Tech Stack
Categories
About Databricks
Databricks is the Data and AI company. More than 20,000 organizations worldwide — including adidas, AT&T, Bayer, Block, Mastercard, Rivian, Unilever, and over 60% of the Fortune 500 — rely on Databricks to build and scale data and AI apps, analytics and agents. Headquartered in San Francisco with 30+ offices around the globe, Databricks offers a unified Data Intelligence Platform that includes Agent Bricks, Lakeflow, Lakehouse, Lakebase and Unity Catalog. --- Databricks applicants Please apply through our official Careers page at databricks.com/company/careers. All official communication from Databricks will come from email addresses ending with @databricks.com or @goodtime.io (our meeting tool).