8 days ago
Base Salary
$145k - $220k/yr
Responsibilities
- Design and build batch and streaming ingestion pipelines from enterprise source systems into a Databricks lakehouse using Delta Lake.
- Own Bronze-layer ingestion patterns, including raw landing, metadata capture, traceability, and recoverable processing.
- Build Silver-layer pipelines for cleansing, standardization, deduplication, conformance, and quality enforcement.
- Define reusable ingestion and transformation patterns, templates, and engineering standards for multiple domains.
- Implement Databricks catalogs, schemas, service principals, job orchestration, and environment-aware deployment patterns.
- Build and maintain CI/CD pipelines for ingestion code, transformations, workflows, tests, and environment promotion.
- Apply data classification, segregation, access, security, retention, and compliance requirements to pipeline design.
- Develop business-oriented data quality controls and quarantine, flag, or route problematic records and datasets.
- Document source objects, ingestion logic, transformations, quality rules, and known limitations.
- Partner with Analytics Engineers and domain teams to deliver reliable, governed Silver-layer data and support domain onboarding.
Requirements
- 12+ years of data engineering experience, including hands-on ownership of production data pipelines.
- Strong experience with Databricks, Delta Lake, Databricks Workflows or Jobs, Spark, PySpark, and/or Spark SQL.
- Working knowledge of Unity Catalog, including catalogs, schemas, tables, lineage, and access control concepts.
- Experience with batch, CDC, and/or streaming ingestion patterns and their operational trade-offs.
- Experience with CI/CD, deployment automation, version control, testing, and controlled promotion across environments.
- Strong SQL skills and grounding in data modeling fundamentals.
- Experience understanding business and regulatory context and applying data classification, segregation, security, retention, or compliance requirements in data engineering workflows.
- Ability to design pipelines around data sensitivity, ownership, permitted use, and downstream impact.
- Ability to shape standards and reusable patterns in a fast-moving platform build.
- Preferred: Databricks Data Engineer Associate or Professional certification.
- Preferred: experience with Unity Catalog structures and access models, infrastructure-as-code, and deployment automation for data platforms.
- Preferred: experience in defense, aerospace, financial services, healthcare, or other regulated environments.
- Preferred: experience enabling multiple business domains on a shared governed data platform.
Benefits
- Full-time regular employees receive pay within the listed range, bonus, benefits, and equity; the posting does not state the compensation figures.
- Temporary employees receive pay within the listed range and a temporary benefits package applicable after 60 days of employment.
- Offers are contingent on a cleared background and possible reference check.
- Military fellows and part-time employees are not eligible for benefits.
- Shield AI offers equal employment opportunity and accommodations for disabilities or special needs.
Tech Stack
Categories
Data Engineering
About Shield AI
Shield AI builds autonomous systems for military and national security customers, combining its Hivemind autonomy software with V-BAT and X-BAT unmanned aircraft and Aechelon simulation technologies. The privately held company sells hardware, software, and related services to U.S. and allied defense agencies. Founded in 2015 and headquartered in San Diego, it operates across the U.S., Europe, the Middle East, and Asia-Pacific, and its technology is used in operational deployments.