1 month ago
Responsibilities
- Design batch-first data flows that process billions of documents efficiently and add streaming edges where they create meaningful product impact.
- Prevent, detect, and contain bad or late data and execute safe backfills and rollbacks.
- Build instrumentation, alerting, and operational playbooks to keep pipelines healthy.
- Identify bottlenecks, reduce failure modes, and improve availability, freshness, correctness, and related SLI/SLA measures.
- Collaborate with ranking and serving teams to deliver end-to-end improvements.
- Mentor teammates, codify best practices, and set technical direction across multiple pipelines or domains.
Requirements
- Proven ownership of production data pipelines at significant scale.
- Strong production implementation skills in Java and/or Python plus SQL, beyond orchestration or UI configuration.
- Sound judgment on batch versus streaming and where near-real-time updates, such as CDC or event streams, provide material value.
- Experience with Kafka and familiarity with Flink, Flink SQL, and streaming fundamentals.
- A strong data-correctness mindset.
- On-call experience, robust observability practices, and a record of improving availability, freshness, or correctness.
- Experience setting technical direction across multiple pipelines or domains.
- Search indexing exposure, including document or feature pipelines and index build or merge mechanics, is a bonus.
- Experience with Spark, MapReduce, BigQuery, GCP, Iceberg, Hudi, Kafka, or Flink is a bonus.
- Deeper Flink knowledge, including state, checkpointing, and savepoints, is a bonus.
Benefits
- Remote-first, digital-by-default work arrangement with teams distributed across North America and beyond.
- Senior candidates will also be considered based on scope.
Tech Stack
Categories
Data Engineering
About Shopify
Shopify is a leading global commerce company, providing trusted tools to start, grow, market, and manage a retail business of any size. Shopify makes commerce better for everyone with a platform and services that are engineered for reliability, while delivering a better shopping experience for consumers everywhere. Shopify powers millions of businesses in more than 175 countries and is trusted by brands such as Allbirds, Gymshark, PepsiCo, Staples, and many more. Find all our jobs here: www.shopify.com/careers