Senior Databricks Platform Engineer
General Dynamics Information Technology3 days ago
Remote, United StatesSenior
Base Salary
$187k - $252k/yr
Responsibilities
- Design, configure, administer, maintain, and govern the enterprise Databricks Lakehouse Platform, including workspaces, Unity Catalog, and scalable data architectures.
- Administer and optimize Databricks compute, clusters, SQL warehouses, serverless capabilities, workload management, autoscaling, auto-termination, and right-sizing.
- Develop and optimize data pipelines, ETL/ELT processes, and ingestion frameworks using Spark, Delta Lake, Delta Live Tables, and Databricks Workflows.
- Create and govern Unity Catalog catalogs, schemas, tables, roles, groups, RBAC, access controls, and data governance policies.
- Automate platform provisioning, configuration, testing, and deployments using Terraform, Python, SQL, Databricks Asset Bundles, and Databricks APIs.
- Implement monitoring, logging, alerting, observability, capacity planning, performance tuning, scalability testing, and troubleshooting.
- Implement security, encryption, networking, auditing, compliance, high availability, disaster recovery, incident response, and root-cause analysis controls.
- Support platform upgrades, patching, versioning, release management, cost optimization, DBU forecasting, financial reporting, and TCO analysis.
- Integrate Databricks with enterprise data sources, governance tools, BI platforms, and AI/ML environments.
- Establish platform standards, reusable engineering patterns, reference architectures, runbooks, and technical documentation.
- Collaborate with architecture, security, governance, and data engineering teams while providing technical guidance and mentorship to platform users.
Requirements
- A MA/MS degree and 12+ years of general experience plus 10+ years of specialized experience, or qualifying equivalent experience in lieu of the degree.
- Strong hands-on experience implementing and administering the Databricks Lakehouse Platform, including Unity Catalog, cluster and SQL warehouse management, and job/workflow orchestration.
- Experience with Databricks performance engineering, scalability testing, workload management, Spark performance tuning, and SQL.
- Experience with observability and performance tools including Grafana, Prometheus, Datadog, CloudWatch, and Elastic.
- Experience with CI/CD pipelines using Jenkins, GitLab CI, GitHub Actions, or Azure DevOps.
- Proficiency in Python and Bash for automation, testing, and platform scripting, with strong SQL skills.
- Experience operating Databricks in AWS-based or hybrid cloud environments.
- Familiarity with Docker, Kubernetes, microservices patterns, telemetry, logging, traceability, root-cause analysis, version control, build and release processes, and infrastructure as code.
- Knowledge of data ingestion, ETL/ELT, lakehouse architectures, distributed data processing, cloud security, IAM, networking, access control, and resource governance.
- Familiarity with federal security, compliance, and audit processes including ATO and FedRAMP.
- Preferred certifications include Databricks, AWS, Docker, DevSecOps, data management, and ITIL certifications.
Benefits
- Remote work from any location with no travel required and a 40-hour scheduled work week.
- Medical, dental, vision, health savings account, and 401(k) plan options with company matching.
- Flexible work weeks where possible, paid vacation, sick and personal leave, holidays, parental, military, bereavement, and jury-duty leave.
- Typically 15 days of paid leave and 10 paid holidays per calendar year for new employees, prorated based on hire date.
- Up to 160 hours of paid family leave in a rolling 12-month period for eligible employees.
- Short- and long-term disability, life, accidental death and dismemberment, personal accident, critical illness, and business travel and accident insurance.
Tech Stack
Apache SparkAWSBashDatabricksDatadogDockerGitHub ActionsGitLab CI/CDGrafanaJenkinsKubernetesPrometheusPythonSQLTerraform
Categories
Data EngineeringDevOps
About General Dynamics Information Technology
General Dynamics Information Technology (GDIT) delivers enterprise IT, systems integration, cybersecurity, cloud, and software engineering services to U.S. federal agencies, including defense and intelligence organizations. It provides contract-based professional and managed services ranging from network operations and unified communications to training and simulation. GDIT is a business unit of General Dynamics (NYSE: GD) and is headquartered in Falls Church, Virginia.