about 3 hours ago
Responsibilities
- Own the end-to-end lifecycle of data pipelines across more than 15 regions, from design through production.
- Set technical direction for data pipeline development and operations.
- Write and optimize distributed processing jobs for massive datasets.
- Build and maintain pipelines feeding the product access layer for threat intelligence, assets, and detections.
- Evolve the underlying data models.
- Design data-health frameworks with conditional checks and comprehensive observability.
- Lead observability efforts across all pipelines.
- Improve engineering quality through design reviews, code reviews, and mentorship.
- Participate in on-call operations and automate operational workflows.
Requirements
- 7+ years of experience building and operating production data pipelines at scale, including designing and running systems independently.
- Deep Python 3.10+ experience, including Pydantic, asynchronous programming, decorators, packaging, and profiling.
- Production-scale Apache Airflow DAG authoring experience, including dynamic task mapping, deferrable operators, SLAs, and MWAA or self-hosted operations and upgrades.
- Strong PySpark job-authoring and optimization experience, including partitioning, shuffle and skew mitigation, join strategy, and reading Catalyst plans, with EMR or Databricks.
- Deep Snowflake experience, including Snowpark, schema and clustering design, query tuning, warehouse sizing, and cost attribution.
- AWS experience with S3, SQS/SNS, EMR, IAM, and infrastructure-as-code.
- Experience designing data quality frameworks using Great Expectations or an equivalent tool.
- Experience with distributed tracing, structured logging, and metrics for observability.
- Experience with artifact packaging and staged deployments across multiple environments and regions.
- Strong testing discipline using pytest, moto, and integration tests against real or mocked cloud services.
Benefits
- Hybrid work arrangement.
- On-call responsibilities with automation tools to reduce operational burden.