about 4 hours ago
Base Salary
$196k - $230k/yr
Responsibilities
- Design and build scalable platform primitives for Spark and Airflow.
- Lead the migration and modernization of Spark workloads to serverless Databricks and Delta Lake.
- Optimize compute resource utilization and efficiency across large-scale distributed systems.
- Collaborate with internal teams to deliver a seamless, self-serve data processing experience.
- Implement advanced metadata management and access controls to improve platform reliability and governance.
Requirements
- Extensive experience with large-scale Spark and Databricks or similar platform infrastructure.
- Deep expertise in data orchestration using Airflow for complex job lifecycle management.
- Proven track record with lakehouse fundamentals, including S3-based data lakes and Delta Lake.
- Familiarity with query and serving infrastructure such as Trino, Pinot, or Hive Metastore.
- Ability to own multi-team platform reliability and developer experience initiatives.
Benefits
- Challenging, high-impact work to grow your career.
- Performance driven compensation with bonus programs and equity ownership.
- 100% paid health insurance for employees with 90% coverage for dependents.
- Access to the best AI tools and continuous skill-building opportunities.
- Flexible benefits spending account for wellness and learning.
- Employer-paid life and disability insurance, fertility benefits, and mental health support.
- Time off for company holidays, paid time off, sick time, and parental leave.
- Exceptional office experience with catered meals and comfortable workspaces.
Tech Stack
Apache AirflowApache SparkDatabricks