Databricks Data Engineer, Senior
Booz Allen Hamilton2 days ago
Base Salary
$78k - $176k/yr
Responsibilities
- Design, build, deploy, maintain, and optimize end-to-end data pipelines and ETL processes.
- Develop and optimize Databricks notebooks, clusters, job orchestration, Delta Lake data flows, and Unity Catalog usage.
- Write Python and PySpark ETL scripts and custom transformations.
- Design schemas and data models, including dimensional and relational models within Bronze, Silver, and Gold medallion architectures.
- Implement comprehensive logging, error handling, troubleshooting, partitioning, bucketing, and caching to improve data processing and query performance.
- Use AWS data services to build scalable data processing and analytics platforms.
- Support metadata management, data lineage, data governance, privacy, security, and compliance requirements.
- Collaborate with cross-functional teams and provide technical leadership and mentorship to analysts, data engineers, developers, and data consumers.
- Explain complex technical concepts and architectural tradeoffs to non-technical audiences.
Requirements
- At least 8 years of experience designing, developing, deploying, and managing ETL processes and data pipelines.
- At least 3 years of experience building, debugging, and optimizing Databricks data pipelines using notebooks, clusters, job orchestration tools, Delta Lake, Databricks Delta, and Unity Catalog.
- At least 3 years of experience writing Python and PySpark code for ETL scripts and custom transformations.
- Experience with data modeling, medallion architecture, performance optimization, and efficient analytics data structures.
- Experience with AWS Glue, Amazon S3, and Amazon RDS.
- Bachelor’s degree in Computer Science, Computer Engineering, Engineering, Mathematics, Physics, Information Systems, or Information Technology.
- Ability to obtain and maintain a Public Trust or Suitability/Fitness determination based on client requirements.
- Preferred experience with AWS Lambda, Amazon CloudWatch, AWS Glue Data Catalog, or Collibra.
- Preferred knowledge of data governance, data lineage, GDPR, CCPA, data quality, privacy, and security.
- Ability to explain complex technical concepts and architectural tradeoffs to non-technical audiences.
Benefits
- Hybrid work arrangement with work expected at a Booz Allen or customer facility according to leadership expectations and role needs.
- Health, life, disability, financial, and retirement benefits.
- Paid leave, professional development, tuition assistance, work-life programs, and dependent care benefits.
- Recognition awards and access to Booz Allen employee benefit programs for eligible full-time and part-time employees working at least 20 hours per week.
- Applicants selected may be subject to a government investigation and U.S. government eligibility requirements.
About Booz Allen Hamilton
Booz Allen Hamilton builds and integrates software, analytics, AI, cloud, and cybersecurity solutions while providing engineering and consulting services, primarily for U.S. defense, intelligence, and civil government clients, with select commercial partnerships. Its business is largely government contracting and professional services delivering mission systems and cyber defense. Founded in 1914 and headquartered in McLean, Virginia, Booz Allen is a public company traded on the NYSE (ticker: BAH).