3 days ago
Reston, VA, USASenior
Base Salary
$99k - $203k/yr
Responsibilities
- Design, configure, and operate Azure Databricks workspaces, clusters, jobs, workflows, catalogs, storage credentials, external locations, and service principals.
- Build and optimize batch and real-time ingestion and transformation pipelines using Python, PySpark, SQL, Delta Lake, Delta Live Tables, Auto Loader, and Apache Spark.
- Implement medallion architecture, dimensional models, change data capture, schema evolution, data quality checks, retry and rollback mechanisms, and reproducible processing patterns.
- Configure Unity Catalog governance, including role-based access, lineage, retention, catalog and schema provisioning, and secure integration with ADLS Gen2 and other approved storage.
- Develop frameworks and interfaces that provide AI and analytics teams with governed training, inference, and reporting data.
- Automate Databricks deployments and environment configuration using Terraform, Databricks Asset Bundles, GitHub Actions, Azure DevOps, or comparable tools.
- Integrate Databricks with Azure Data Factory, Kafka, Airflow, APIs, Snowflake, Synapse, and other enterprise data services.
- Monitor pipeline, cluster, and workspace health; diagnose performance and data issues; and support root cause analysis and operational reliability.
- Support security, ATO, vulnerability remediation, business continuity, disaster recovery, and operational readiness for Databricks environments and data products.
- Participate in Agile planning, code reviews, demos, and retrospectives, and document architecture, runbooks, data models, governance decisions, and deployment procedures.
Requirements
- BA/BS/MS in Computer Science, Engineering, Data Science, or equivalent experience.
- More than 5 years of data engineering experience, including at least 3 years supporting Azure Databricks and related Azure data services.
- Expertise in Apache Spark, Delta Lake, Python, SQL, and Scala, with hands-on experience designing and optimizing large-scale ETL/ELT pipelines.
- Experience with Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, Databricks workflows, data modeling, and big data technologies.
- Experience managing Databricks workspaces, clusters, compute resources, security controls, access management, and cost optimization.
- Experience implementing CI/CD or infrastructure as code for data platforms using Git and tools such as Terraform, Databricks Asset Bundles, GitHub Actions, or Azure DevOps.
- Experience with Agile delivery, data governance, data security, performance troubleshooting, and communication with technical and business stakeholders.
- Ability to satisfy FDIC personnel eligibility and background-investigation requirements.
- U.S. citizenship, U.S. residency, and authorization to work in the U.S. are required.
- Preferred qualifications include Databricks certifications, streaming and real-time data experience, enterprise data platform experience, Generative AI and ML workload experience, cross-cloud architecture experience, BCDR and security-controls experience, federal compliance experience, and experience with Docker, Kubernetes, REST APIs, FastAPI, or Node.js.
Benefits
- Remote position with all work performed in the United States.
- Full-time employment; pay is based on experience, skills, certifications, competencies, location, education, and applicable contract provisions.
- Equal opportunity employer with reasonable accommodations available for disabilities, disabled veterans, and sincerely held religious beliefs.
Tech Stack
Apache AirflowApache KafkaApache SparkAzureDatabricksDockerFastAPIGitGitHub ActionsKubernetesMicrosoft SQL ServerMLflowNode.jsPythonScalaSnowflakeSonarQubeSplunkSQLTerraform
Categories
Data Engineering
About ICF
ICF is a global consulting and technology services firm serving government and commercial clients across domains such as energy, health, climate, and digital modernization. It delivers advisory, program implementation, analytics, and software-enabled solutions on platforms like ServiceNow. Founded in 1969 and headquartered in Reston, Virginia, ICF is a public company traded on NASDAQ and works extensively with U.S. federal, state and local agencies as well as regulated industries.