Principal Engineer, Compute Fleet Management
Databricks10 months ago
Base Salary
$264k - $322k/yr
Responsibilities
- Provision and pool billions of cloud resources across AWS, Azure, and GCP for workload performance and efficiency.
- Architect horizontally scalable systems resilient to zonal and cloud account-level failures.
- Lead development of the lowest-dependency systems used to bootstrap and manage Databricks' compute platform.
- Drive 99.99% availability for batch and serving workloads.
- Increase compute utilization to 60% or higher while maintaining tolerance for cloud failures.
- Architect and enforce strong security and performance isolation across diverse customer workloads.
- Own complex strategic engineering initiatives and coordinate execution across teams and organizational boundaries.
Requirements
- Seasoned Principal Engineer with a track record of building and operating large-scale, mission-critical infrastructure systems in production.
- Deep hands-on experience developing and operating high-scale distributed systems on at least one major public cloud.
- Experience leading complex, cross-team, cross-layer, multi-quarter strategic engineering initiatives from concept through execution.
- Proven ability to establish technical direction, drive consensus, and lead large technical efforts across organizational boundaries.
- Exceptional planning, project-tracking, and management of complex cross-organizational dependencies.
- Experience managing and scaling a massive GPU fleet for AI/ML workloads is highly desirable.
- Experience developing and operating large-scale distributed systems across AWS, Azure, and GCP is highly desirable.
Benefits
- Comprehensive benefits and perks, with region-specific details available through Databricks' benefits site.
Tech Stack
About Databricks
Databricks builds a cloud-based data and AI platform centered on the lakehouse architecture, combining data engineering, analytics, and machine learning with Apache Spark, Delta Lake, and MLflow. It sells subscriptions and cloud services to enterprises that need to unify data pipelines and develop large-scale AI and analytics. Founded in 2013 by the creators of Apache Spark and headquartered in San Francisco, the company is privately held and serves organizations across many industries.