AWS Databricks Engineer
Credence Management Solutions, LLC1 day ago
McLean, VA, USAMid Level
Base Salary
$155k - $185k/yr
Responsibilities
- Design, build, and optimize scalable data pipelines and processing workflows with Databricks, Apache Spark, PySpark, and Delta Lake.
- Implement AWS-based data lakehouse solutions using medallion architecture and services including S3, Glue, RDS, Redshift, and Lambda.
- Develop and optimize ETL/ELT and data transformation workflows using SQL, Python, and PySpark.
- Implement data quality, cataloging, lineage, governance, and access-control practices, including Unity Catalog where applicable.
- Automate data platform deployments and workflows using Infrastructure as Code, CI/CD pipelines, Git-based development, and AWS-native deployment tools.
- Collaborate with technical teams and mission stakeholders to deliver production-ready data solutions.
- Produce maintainable, documented code and support code reviews, testing, troubleshooting, and technical design documentation.
Requirements
- U.S. citizenship and the ability to obtain and maintain a Secret clearance are required.
- Bachelor’s or master’s degree in Computer Science, Data Engineering, Information Systems, Engineering, or a related technical field.
- Hands-on experience in data engineering, cloud engineering, or related technical roles.
- Production experience with Databricks, Apache Spark, PySpark, Delta Lake, SQL, and Python.
- Strong experience with AWS data services including S3, Glue, RDS, Redshift, Lambda, IAM, and CloudWatch.
- Understanding of ETL/ELT, data lake and lakehouse architectures, data modeling, large-scale data processing, cloud security, identity and access management, encryption, and secure data handling.
- Experience developing, scheduling, monitoring, and troubleshooting production data pipelines and workflows.
- Experience with Git and CI/CD platforms such as GitLab, GitHub Actions, Jenkins, or AWS CodePipeline.
- Strong communication skills and a client-oriented approach.
- Preferred qualifications include Databricks Lakehouse and medallion architecture experience, Unity Catalog, Delta Live Tables, Databricks Workflows, Pandas, Pydantic, Infrastructure as Code tools, AWS or Databricks certifications, and federal cybersecurity or regulated-environment experience.
Benefits
- Medical, dental, and vision healthcare plans.
- 401k and IRA retirement plans.
- Basic, voluntary, and AD&D life insurance.
- Paid vacation, sick leave, and public holidays.
- Maternity and paternity family leave.
- Short- and long-term disability coverage.
- Training and development opportunities.
- Wellness resources.
- Full-time, on-site position in McLean, United States.
Categories
Data Engineering
About Credence Management Solutions, LLC
Credence Management Solutions, LLC provides IT consulting, engineering, and mission support to U.S. federal agencies, with emphasis on defense, intelligence, health, and humanitarian programs. It builds and operates secure AI, cloud, cyber, and DevSecOps solutions and delivers program management and systems engineering under government contracts. The privately held firm is headquartered in McLean, Virginia, and supports customers such as the U.S. Air Force at Wright-Patterson Air Force Base.