Senior Data Engineer
Hack The Box2 months ago
Remote, United States or New York, NY, USASenior
Base Salary
$140k - $160k/yr
Responsibilities
- Design and build batch and streaming pipelines on Dataflow, Pub/Sub, and Kafka feeding BigQuery, Bigtable, and ClickHouse.
- Drive migration from Snowflake to the GCP-native data stack and retire legacy or shadow pipelines.
- Own Airflow orchestration, including SLAs, retries, and data quality gates.
- Model data for analytics and ML, including feature pipelines for training and low-latency online inference.
- Partner with ML engineers on feature stores, data drift monitoring, and retraining workflows.
- Improve data quality, reliability, observability, and cost efficiency across data systems.
- Capture stakeholder requirements and translate them into pragmatic data products.
- Identify and integrate new data sources, including third-party SaaS sources.
Requirements
- Strong data modelling and warehouse architecture skills, including dimensional modelling, event-driven patterns, and lakehouse patterns.
- Hands-on experience with GCP data services, with BigQuery required and Pub/Sub, Dataflow, Bigtable, and Cloud Composer valued.
- Production experience building streaming pipelines with Dataflow/Beam, Flink, or Spark Structured Streaming using Kafka and/or Pub/Sub.
- Strong Python and SQL skills and experience writing production-quality code.
- Production experience with ClickHouse or another columnar OLAP engine.
- Experience with Airflow or equivalent workflow orchestration tools such as Prefect or Dagster.
- Experience with dbt or an equivalent transformation framework.
- Experience migrating from legacy warehouses such as Snowflake, Redshift, or Synapse to cloud-native stacks is a plus.
- Working knowledge of production ML, including feature engineering, feature stores, model deployment, drift monitoring, and retraining.
- Experience with Docker and Kubernetes, plus infrastructure-as-code and CI/CD practices.
- CDC tooling experience with Datastream or Debezium and Vertex AI or Feature Store experience are bonuses.
Benefits
- Fully remote work in the United States.
- Base compensation of $140,000–$160,000.
- 100% employer-paid employee medical, dental, and vision coverage.
- 401(k) with employer match, employer-paid life and AD&D insurance, supplemental life insurance, and short- and long-term disability coverage.
- Healthcare and dependent care FSA, paid parental leave, and 25 annual leave days.
- Home office allowance, state-of-the-art equipment, and a dedicated training and professional development budget with conference participation.
- Full access to Hack The Box lab offerings.
Tech Stack
Amazon RedshiftApache AirflowApache BeamApache FlinkApache KafkaApache SparkClickHousedbtDockerGitHub ActionsGoogle BigQueryGoogle Cloud PlatformKubernetesPythonSnowflakeSQL
Categories
Data Engineering