Data Engineer - Foundational
Harmattan AI8 months ago
Paris, FranceSenior
Responsibilities
- Build ETL/ELT pipelines to extract, decode, and store raw EO and IR video in optimized formats including WebDataset, TFRecords, and Parquet.
- Develop algorithms to temporally and spatially synchronize EO and IR frames for model training.
- Architect storage-to-GPU pipelines for multi-node training clusters and maintain greater than 90% GPU utilization.
- Write and optimize distributed data-processing jobs using Apache Spark, Ray, or Apache Beam.
- Implement automated data-quality checks for corrupted or blank frames and maintain reproducible training runs through versioning and lineage tracking.
- Evaluate and implement storage solutions such as MinIO and S3 tiering to optimize dataset cost and latency.
Requirements
- Bachelor’s or master’s degree in Computer Science, Software Engineering, or Distributed Systems is highly preferred.
- 5–6+ years of experience building and maintaining terabyte-scale pipelines for unstructured data such as video, images, or point clouds.
- Deep knowledge of operating systems, networking, and parallel computing.
- Demonstrated experience maximizing multi-node GPU utilization and optimizing data loaders for PyTorch or JAX.
- Strong command of distributed computing tools including Spark, Ray, and Beam.
- Experience with ML data-versioning tools such as DVC, Apache Iceberg, or Pachyderm.
- Systems-oriented approach, strong ownership, and ability to automate solutions for messy data.
Benefits
- The role is based in Paris.
- The position supports Harmattan AI’s mission of delivering ethical, high-impact autonomous defense technology.
Tech Stack
Categories
Data Engineering
About Harmattan AI
Harmattan AI builds autonomous defense systems—air and multi-domain robots and the software that coordinates them—for military and government customers. Its products span autonomy stacks for high-speed flight, guidance and control, computer vision, and radar-enabled sensing, delivered as deployable platforms and integrated software. Founded in 2024 and headquartered in Paris, the privately held company operates as a defense prime serving allied forces.