4 months ago
Manassas, VA, USASenior
Responsibilities
- Implement real-time data pipelines with MQTT and Redpanda for stream processing.
- Implement offline data pipelines using Dagster for batch processing.
- Parse and process binary message formats from various data sources.
- Build data warehouses using Postgres, Apache Iceberg, Parquet, and S3.
- Design data models that allow for high-performance queries.
- Validate and normalize data sources.
- Improve local development and CI/CD using modern tooling and GitHub Actions.
Requirements
- Strong software engineering skills with a focus on code quality and performance.
- Expertise in time-series data processing and analysis.
- Proficiency in Python and Rust for data engineering workflows.
- Experience with binary message parsing and data formats.
- Knowledge of distributed systems, streaming architectures, and batch processing patterns.
- Hands-on experience with batch orchestrators like Dagster/Airflow and streaming platforms like Redpanda/Kafka.
- Current or active U.S. Secret clearance.
Benefits
- Unlimited PTO for work-life balance.
- Dedicated sick time for health and well-being.
- Comprehensive health and benefits including medical, dental, and vision coverage.
- 11 paid holidays throughout the year.
- Professional development opportunities for skill growth.
- Collaborative environment with direct access to leadership.
- 401(k) with company match and performance bonuses.
Tech Stack
Categories
Data Engineering
About Spear AI
Spear AI builds AI/ML-driven hardware, software, and data solutions for national security, including unmanned maritime sensor systems designed for operational deployment across classification levels. The privately held company sells products and implementation services to defense agencies and contractors, with work spanning reinforcement learning, acoustic ML, and cloud architectures. Founded in 2020 and headquartered in Washington, DC.