Staff Software Engineer (Data Platform)
Databricks2 months ago
Bengaluru, IndiaStaff+
Responsibilities
- Design and operate the Databricks metrics store for shared metric aggregation, quality, introspection, and query performance.
- Design and run the cross-company Data Intelligence Platform containing business and product metrics.
- Develop tooling and infrastructure for operating Databricks at scale across clouds, geographies, and deployment types.
- Build the company-wide base ETL framework and reliable data pipelines.
- Create CI/CD processes, pipeline test frameworks, data-quality tooling, and infrastructure-as-code solutions.
- Establish APIs and conventions for telemetry, debug, feature, and audit-event data.
- Provide technical leadership and long-term product direction in partnership with engineering teams.
- Represent Databricks at academic and industry conferences and events.
Requirements
- 12+ years of industry experience.
- 4+ years of experience building large-scale distributed systems.
- 5+ years providing technical leadership on large projects involving ETL frameworks, metrics stores, infrastructure management, or data security.
- Experience building, shipping, and operating reliable multi-geo data pipelines at scale.
- Experience operating workflow or orchestration frameworks such as Airflow and dbt, or comparable commercial tools.
- Experience with large-scale messaging systems such as Kafka or RabbitMQ, or comparable commercial systems.
- Strong cross-functional communication and consensus-building skills.
- Passion for data infrastructure and improving data accessibility for others.
Benefits
- Databricks offers comprehensive benefits and perks, with specific details varying by region.
Tech Stack
Categories
Data Engineering
About Databricks
Databricks builds a cloud-based data and AI platform centered on the lakehouse architecture, combining data engineering, analytics, and machine learning with Apache Spark, Delta Lake, and MLflow. It sells subscriptions and cloud services to enterprises that need to unify data pipelines and develop large-scale AI and analytics. Founded in 2013 by the creators of Apache Spark and headquartered in San Francisco, the company is privately held and serves organizations across many industries.