Data Engineer - AWS/Databricks
Acuity, Inc.2 years ago
Reston, VA, USASenior
Base Salary
$120k - $150k/yr
Responsibilities
- Build and maintain scalable PySpark data pipelines in Databricks for ingestion, transformation, and enrichment.
- Design and optimize Delta Lake tables and Spark SQL/DataFrame workflows for large-scale data processing.
- Develop ETL and ELT workflows integrating multiple source systems into centralized data warehouse architectures.
- Implement AWS cloud data solutions using S3, Glue, RDS, and IAM.
- Optimize pipeline performance through partitioning, caching, broadcast joins, and adaptive query tuning.
- Deploy and version data engineering assets using Git-integrated workflows and GitLab or Jenkins CI/CD tools.
- Monitor Databricks jobs, clusters, and AWS CloudWatch metrics to identify bottlenecks and optimize cost and performance.
- Map legacy source systems and design end-to-end data flows and transformations.
- Implement metadata tagging, data quality validation, audit logging, and lineage tracking.
- Support data access requests and develop reusable data assets and shared notebooks for reporting and analytics.
Requirements
- At least 5 years of experience in data engineering and Agile analytics.
- At least 5 years of experience creating software to retrieve, parse, and process structured and unstructured data.
- At least 2 years of experience building scalable ETL and ELT workflows for reporting and analytics.
- At least 3 years of experience building enterprise cloud data engineering solutions, preferably with AWS and Databricks.
- Experience with data quality, validation frameworks, and storage optimization strategies.
- BA or BS degree.
- Must be a US Citizen with the ability to obtain and maintain US Suitability.
Tech Stack
Categories
Data Engineering
About Acuity, Inc.
Acuity, Inc. is a federal IT consulting firm that designs and implements technology solutions for U.S. government agencies, with practices in IT modernization, cloud, data engineering and analytics, cybersecurity, and hyperautomation. It delivers program management and systems integration services, including AI/ML and secure cloud deployments compliant with NIST and FedRAMP. The company focuses on the U.S. public sector and operates in highly regulated environments supporting defense, intelligence, and civilian missions.