Principal Software Architect - Data Platform
SecurityScorecard24 hours ago
Responsibilities
- Own end-to-end system design for the data platform from ingestion through serving.
- Define service boundaries, schema ownership, data contracts, compatibility rules, and breaking-change processes.
- Design lakehouse table formats, partitioning, schema evolution, compaction, and metadata strategies.
- Architect isolated analytical serving layers for customer queries, internal exploration, and external bulk delivery.
- Set direction for languages and frameworks across the data stack.
- Build data quality, validation, quarantine, freshness, completeness, drift detection, lineage, and metadata capabilities into the platform.
- Design for correctness, reproducibility, backfills, and historical restatements in the ratings pipeline.
- Write technical design reviews, design documents, standards, and decision records that establish architecture direction.
- Review engineering designs across the organization and provide substantive architecture and risk feedback.
- Partner with AI and Front End Architects and mentor senior and staff engineers on data system design.
Requirements
- 10+ years of software or data engineering experience, including significant experience architecting large-scale data platforms.
- Deep expertise in stream and batch processing at scale with Kafka, Flink, Spark, or close equivalents.
- Strong Python and PySpark skills, solid Java for Flink processing, and enough Scala to read and reason about Spark code.
- Production experience designing lakehouse storage with Parquet, Iceberg, partitioning, compaction, and schema evolution.
- Experience architecting OLAP and analytical serving layers using technologies such as ClickHouse, Druid, Pinot, BigQuery, or Snowflake.
- Strong distributed systems knowledge, including delivery semantics, ordering, backpressure, late data, out-of-order data, and pipeline failure modes.
- Experience engineering data quality, contracts, schema enforcement, assertions, observability, and lineage into production systems.
- Experience owning large-scale data migrations while preserving history and correctness through cutover.
- Demonstrated ability to influence technical direction without formal authority and provide rigorous design reviews.
- Strong technical writing, mentorship, and senior individual-contributor experience.
- Preferred: experience with feature stores, vector stores, or retrieval pipelines for ML or LLM systems.
- Preferred: experience with internet-scale scan, telemetry, or observability data or a cybersecurity background.
- Preferred: experience reducing platform costs through storage tiering, query governance, or compute right-sizing.
Benefits
- Competitive salary, stock options, health benefits, unlimited PTO, parental leave, and tuition reimbursement, with offerings specific to each country.
- The posting states that the estimated total compensation range is $240,000-$300,000, including base plus bonus; actual compensation varies by factors such as skills, qualifications, experience, and affordability.
Tech Stack
Apache FlinkApache KafkaApache SparkAWSC++ClickHouseGoogle BigQueryHelmJavaKubernetesNode.jsPostgreSQLPythonReactScalaSnowflakeTerraformTypeScript
Categories
Data Engineering
About SecurityScorecard
SecurityScorecard builds a SaaS platform that issues cybersecurity ratings and tools for third-party risk and vendor due diligence, used by enterprises, insurers, and public-sector bodies. Customers subscribe to monitor their own posture and that of suppliers, map findings to questionnaires, and support board reporting and underwriting decisions. Founded in 2013 and headquartered in New York, it continuously rates over 12 million companies and serves more than 25,000 organizations.