Principal Software Architect - Data Platform
SecurityScorecard7 hours ago
Responsibilities
- Own the end-to-end system design of the data platform from ingestion through the serving layer.
- Define service boundaries, data contracts, schema ownership, compatibility rules, and breaking-change processes.
- Design lakehouse storage, including table formats, partitioning, schema evolution, compaction, and metadata growth.
- Architect isolated analytical serving layers for low-latency customer queries, internal exploration, and bulk partner delivery.
- Set direction for languages and frameworks in the data stack.
- Build data quality and observability into the platform through validation, quarantine paths, SLOs, drift detection, lineage, and metadata.
- Design correctness and reproducibility into ratings pipelines, including backfills and historical restatements.
- Write Technical Design Reviews, design documents, standards, and decision records that establish architecture direction.
- Review engineering designs and provide substantive feedback on architecture and risk.
- Partner with AI and Front End Architects and mentor senior and staff engineers on data system design.
Requirements
- 10+ years of software or data engineering experience, including significant experience architecting large-scale data platforms.
- Deep expertise in stream and batch processing at scale with Kafka, Flink, and Spark or close equivalents.
- Strong Python and PySpark, solid Java for Flink stream processing, and enough Scala to read and reason about Spark code.
- Production experience designing lakehouse storage with Parquet, Iceberg, partitioning, compaction, and schema evolution.
- Experience architecting OLAP and analytical serving layers using systems such as ClickHouse, Druid, Pinot, BigQuery, or Snowflake.
- Strong distributed-systems knowledge covering delivery semantics, ordering, backpressure, late and out-of-order data, and pipeline failure modes.
- Experience engineering data quality, contracts, observability, assertions, schema enforcement, and lineage into code.
- Experience owning a large-scale data migration while preserving history and correctness through cutover.
- Demonstrated ability to influence technical direction without authority and provide rigorous design reviews.
- Strong technical writing and mentorship experience, including TDRs, design documents, and decision records.
- Comfort operating as a senior individual contributor through prototyping and technical credibility.
- Preferred: experience with feature stores, vector stores, retrieval pipelines, or data layers for ML or LLM systems.
- Preferred: internet-scale scan, telemetry, or observability data experience, cybersecurity background, or experience reducing platform costs through storage tiering, query governance, or compute right-sizing.
Benefits
- Competitive salary, stock options, health benefits, unlimited PTO, parental leave, tuition reimbursement, and additional country-specific benefits.
- The posting states that the estimated total compensation range is $270,000-$330,000, including base salary and bonus.
- The posting does not specify a remote, hybrid, or in-office work arrangement.
Tech Stack
Apache FlinkApache KafkaApache SparkAWSC++ClickHouseGoogle BigQueryHelmJavaKubernetesNode.jsPostgreSQLPythonReactScalaSnowflakeTerraformTypeScript
Categories
Data Engineering
About SecurityScorecard
SecurityScorecard builds a SaaS platform that issues cybersecurity ratings and tools for third-party risk and vendor due diligence, used by enterprises, insurers, and public-sector bodies. Customers subscribe to monitor their own posture and that of suppliers, map findings to questionnaires, and support board reporting and underwriting decisions. Founded in 2013 and headquartered in New York, it continuously rates over 12 million companies and serves more than 25,000 organizations.