Staff Software Engineer - AI Research Infrastructure
Databricks5 months ago
Base Salary
$199k - $270k/yr
Responsibilities
- Design and implement infrastructure for large-scale experiments, data processing, and model training across HPC clusters, GPU fleets, or cloud-based systems.
- Build job submission, scheduling, monitoring, and experiment-management abstractions that enable rapid researcher iteration.
- Create research developer tooling, CI and testing infrastructure, and workflows that reduce experiment iteration time.
- Influence the long-term roadmap for research computation and how AI Research trains, evaluates, and ships models.
- Partner with research scientists, ML engineers, and platform teams to create robust, repeatable pipelines.
- Mentor engineers working on compute, infrastructure, and AI systems.
Requirements
- Bachelor’s, master’s, or PhD in Computer Science or a related field.
- At least 5 years of software engineering experience, including substantial experience with large-scale distributed systems or infrastructure.
- Deep experience building and operating distributed systems, data pipelines, or large-scale backend services, ideally involving GPUs, clusters, or major cloud providers.
- Proficiency in one or more systems programming languages, such as C++, Rust, Go, Java, or Scala, with the ability to design, implement, and debug complex services.
- Experience building or significantly contributing to cluster schedulers, resource managers, or large-scale job orchestration systems such as Kubernetes, Slurm, Ray, or comparable internal systems.
- Understanding of modern ML training and inference workflows, including distributed training, model parallelism, fine-tuning, and evaluation.
- Experience taking complex systems from prototype to stable, well-owned services with strong operational reliability.
- Clear communication skills and the ability to translate between research needs and infrastructure realities.
Benefits
- Comprehensive employee benefits and perks, with region-specific offerings; annual performance bonus and equity may be available.
Tech Stack
About Databricks
Databricks builds a cloud-based data and AI platform centered on the lakehouse architecture, combining data engineering, analytics, and machine learning with Apache Spark, Delta Lake, and MLflow. It sells subscriptions and cloud services to enterprises that need to unify data pipelines and develop large-scale AI and analytics. Founded in 2013 by the creators of Apache Spark and headquartered in San Francisco, the company is privately held and serves organizations across many industries.