Databricks Data Engineer, Lead
Booz Allen Hamilton22 days ago
Base Salary
$78k - $176k/yr
Responsibilities
- Design, build, deploy, maintain, troubleshoot, and optimize end-to-end ETL processes and data pipelines.
- Develop Databricks pipelines using notebooks, clusters, job orchestration tools, Delta Lake, Databricks Delta, and Unity Catalog.
- Write Python and PySpark code for ETL scripts and custom transformations.
- Design data models and schemas, including dimensional and relational models within a Bronze, Silver, and Gold medallion architecture.
- Use AWS services including AWS Glue, Amazon S3, and Amazon RDS to support data processing and platforms.
- Improve ETL and query performance through partitioning, bucketing, caching, and troubleshooting.
- Ensure data security, governance, compliance, lineage, privacy, and data quality.
- Provide technical leadership and mentorship to analysts, data engineers, developers, and data consumers.
- Assess, design, build, and maintain scalable client platforms and explain complex technical concepts and architectural tradeoffs.
Requirements
- 8+ years of experience designing, developing, deploying, and managing ETL processes and data pipelines with logging and error handling.
- 3+ years of experience building, debugging, and optimizing pipelines using Databricks notebooks, clusters, and job orchestration tools.
- 3+ years of experience writing Python and PySpark code for ETL scripts and custom transformations.
- Experience with data modeling, schemas, dimensional and relational models, and medallion architecture.
- Experience with AWS services including AWS Glue, Amazon S3, and Amazon RDS.
- Bachelor's degree in Computer Science, Computer Engineering, Engineering, Mathematics, Physics, Information Systems, or Information Technology.
- Ability to obtain and maintain a Public Trust or Suitability/Fitness determination based on client requirements.
- Preferred experience with AWS Lambda, CloudWatch, AWS Glue Data Catalog, or Collibra.
- Preferred knowledge of data governance, data lineage, GDPR, CCPA, data quality, privacy, and security.
Benefits
- Health, life, disability, financial, and retirement benefits.
- Paid leave, professional development, tuition assistance, work-life programs, and dependent care.
- Recognition awards for exceptional performance and demonstration of company values.
- Full-time and part-time employees working at least 20 hours per week on a regular basis are eligible for the applicable Booz Allen benefits programs.
- Applicants selected may be subject to a government investigation and must meet applicable U.S. government client eligibility requirements.
About Booz Allen Hamilton
Booz Allen Hamilton builds and integrates software, analytics, AI, cloud, and cybersecurity solutions while providing engineering and consulting services, primarily for U.S. defense, intelligence, and civil government clients, with select commercial partnerships. Its business is largely government contracting and professional services delivering mission systems and cyber defense. Founded in 1914 and headquartered in McLean, Virginia, Booz Allen is a public company traded on the NYSE (ticker: BAH).