Principal Databricks Software Developer
PowerPlan, Inc.18 days ago
Atlanta, GA, USAStaff+
Responsibilities
- Audit high-priority Databricks assets and deliver an evidence-backed assessment of platform risks.
- Design and drive adoption of Databricks compute policies, Spark configurations, notebook patterns, pipeline patterns, and Unity Catalog governance.
- Investigate complex Databricks failures, performance regressions, and cost anomalies to root cause.
- Create reusable defaults, templates, reference architectures, and automation instead of one-off fixes.
- Guide teams on Databricks design decisions and responsible adoption of AI-assisted engineering practices.
- Improve platform reliability, performance, governance, and cost-effectiveness.
Requirements
- 8+ years of experience in data engineering, platform engineering, or related software engineering roles.
- 3+ years of deep hands-on Databricks experience.
- Production experience with Spark internals and performance tuning, including shuffle behavior, query planning, adaptive query execution, memory pressure, skew, and partitioning.
- Strong knowledge of Databricks compute options, cluster policies, instance pools, autoscaling, Photon, and workload right-sizing.
- Experience reviewing and improving notebooks, jobs, and pipelines for runtime, cost, and reliability risks.
- Production experience with Delta Lake, Structured Streaming, Auto Loader, and data quality practices.
- Strong command of Unity Catalog governance, lineage, permission models, and secure handling of sensitive data.
- Experience designing resilient job orchestration with dependency management, retries, timeouts, and alerting.
- Strong communication skills and the ability to translate technical risk into actionable stakeholder language.
- Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent professional experience.
Benefits
- Hybrid work arrangement combining onsite work at the corporate office with work from home.
- Onsite attendance may be required for scheduled office days, team meetings, client meetings, or special events.
Tech Stack
Apache SparkDatabricks
Categories
Data EngineeringDevOps