Senior Software Engineer (Data Platform)
Databricks2 months ago
Bengaluru, IndiaSenior
Responsibilities
- Design and operate a company-wide metrics store supporting shared metric definition, aggregation, introspection, quality, and query performance.
- Design and run the Data Intelligence Platform containing business and product metrics across Databricks.
- Build tooling and infrastructure for Databricks on Databricks across multiple clouds, geographies, and deployment types.
- Design the foundational ETL framework used by company data pipelines.
- Build reliable data pipelines and provide feedback on the design and operation of Databricks and partner data products.
- Create conventions and APIs for telemetry, debug, feature, and audit event data.
- Partner with engineering teams on long-term product vision and requirements.
- Represent Databricks at academic and industrial conferences and events.
Requirements
- 6+ years of industry experience.
- 4+ years of technical leadership on large projects involving ETL frameworks, metrics stores, infrastructure management, or data security.
- Experience building, shipping, and operating reliable multi-geography data pipelines at scale.
- Experience operating workflow or orchestration frameworks such as Airflow and dbt, or equivalent commercial enterprise tools.
- Experience with large-scale messaging systems such as Kafka or RabbitMQ, or equivalent commercial systems.
- Strong cross-functional communication and consensus-building skills.
- Passion for data infrastructure and enabling data access for others.
Benefits
- Databricks offers comprehensive benefits and perks, with specific details varying by region.
Tech Stack
Categories
Data Engineering
About Databricks
Databricks builds a cloud-based data and AI platform centered on the lakehouse architecture, combining data engineering, analytics, and machine learning with Apache Spark, Delta Lake, and MLflow. It sells subscriptions and cloud services to enterprises that need to unify data pipelines and develop large-scale AI and analytics. Founded in 2013 by the creators of Apache Spark and headquartered in San Francisco, the company is privately held and serves organizations across many industries.