Data Engineer - AWS/Databricks
Acuity, Inc.2 years ago
Reston, VA, USAMid Level
Base Salary
$90k - $100k/yr
Responsibilities
- Build and maintain scalable PySpark data pipelines in Databricks notebooks for structured and semi-structured data.
- Design and optimize Delta Lake tables for ACID compliance, partition pruning, schema enforcement, and query performance.
- Develop ETL and ELT workflows integrating multiple source systems into centralized data warehouse architectures.
- Use Spark SQL and DataFrame APIs for business rules, dimensional joins, and aggregations.
- Implement cloud-native data solutions on AWS using S3, Glue, RDS, and IAM.
- Optimize pipeline performance through partitioning, caching, broadcast joins, and adaptive query tuning.
- Deploy and version data engineering assets using Git-integrated workflows and automate deployments with GitLab or Jenkins.
- Monitor Databricks pipelines and AWS infrastructure with native Databricks tools and AWS CloudWatch.
- Conduct legacy-system discovery and design end-to-end data flows and transformations.
- Implement metadata tagging, data quality validation, audit logging, and lineage tracking.
- Support ad hoc data access requests and maintain reusable data assets and shared notebooks.
Requirements
- At least 2 years of experience in data engineering and Agile analytics.
- At least 2 years of experience creating software to retrieve, parse, and process structured and unstructured data.
- One to two years of experience building scalable ETL and ELT workflows for reporting and analytics.
- At least 1 year of experience building enterprise cloud data engineering solutions, preferably with AWS and Databricks.
- Experience with data quality, validation frameworks, and storage optimization strategies.
- BA or BS degree.
- Must be a U.S. citizen with the ability to obtain and maintain U.S. Suitability.
Tech Stack
Categories
Data Engineering
About Acuity, Inc.
Acuity, Inc. is a federal IT consulting firm that designs and implements technology solutions for U.S. government agencies, with practices in IT modernization, cloud, data engineering and analytics, cybersecurity, and hyperautomation. It delivers program management and systems integration services, including AI/ML and secure cloud deployments compliant with NIST and FedRAMP. The company focuses on the U.S. public sector and operates in highly regulated environments supporting defense, intelligence, and civilian missions.