Senior Software Engineer, Data Engineering
Distributed Spectrum22 hours ago
Base Salary
$160k - $200k/yr
Responsibilities
- Architect and build a data lakehouse with ingestion pipelines, open table formats, partitioning, compaction, cataloging, and governance.
- Design and operate vector database infrastructure for embedding storage, similarity search, and retrieval at scale.
- Build lifecycle, retention, lineage, dataset versioning, quality monitoring, and cost-management systems for petabyte-scale storage.
- Develop ML-focused feature, embedding, training-set, evaluation-dataset, and low-latency retrieval pipelines.
- Establish schema and data-contract standards and build self-service tooling for researchers and engineers.
- Own production data-pipeline reliability, performance, observability, and failure handling.
- Mentor engineers and lead technical design across the data domain.
Requirements
- 4–6+ years of software or data engineering experience, including several years designing and operating large-scale production data platforms.
- Deep experience with lakehouse architectures and technologies such as Apache Iceberg, Delta Lake, Hudi, Parquet, Spark, Flink, Trino, or DuckDB.
- Hands-on experience with vector databases and embedding retrieval, including indexing trade-offs and scaling.
- Strong experience with AWS or another major cloud provider, including object storage, managed data services, compute, and large-scale cost optimization.
- Fluency in Python and SQL and experience with orchestration tools such as Airflow, Dagster, Prefect, or Step Functions.
- Experience designing data architectures for ML/AI workloads, including training-data pipelines, feature stores, dataset versioning, or retrieval systems.
- Strong understanding of data modeling, consistency, and batch-versus-streaming trade-offs.
- Experience with high-volume streaming ingestion using Kafka, Kinesis, or Pulsar is preferred.
- Experience with time-series, geospatial, signal, or sensor data is preferred.
- Familiarity with data governance and security requirements in regulated environments is preferred.
- Experience with Rust, Go, or C++ for performance-critical data paths is preferred.
Benefits
- Above-market salary, equity, and benefits package.
- Early Series A equity.
- Excellent health, dental, and vision coverage.
- 401(k) match of up to 4% of salary.
- Flexible PTO.
- Daily office lunches in NYC.
- The role is subject to U.S. ITAR eligibility or required Department of State authorizations.
Tech Stack
Categories
Data Engineering
About Distributed Spectrum
Distributed Spectrum builds software and sensor systems for radio spectrum intelligence, combining signal processing, machine learning, and embedded computing for defense and government operators. Its products collect and analyze RF data and run trained models on low-power devices to detect and interpret critical signals in the field. Founded in 2020 and headquartered in New York, the company is privately held and venture-backed.