10 hours ago
Responsibilities
- Design batch-first data flows processing billions of documents and add streaming edges where they provide product value.
- Prevent, detect, and contain bad or late data while planning and executing safe backfills and rollbacks.
- Build instrumentation, alerting, and playbooks to maintain pipeline health.
- Identify bottlenecks, reduce failure modes, and improve availability, freshness, correctness, and related SLI/SLA measures.
- Collaborate with ranking and serving teams on end-to-end search improvements.
- Mentor teammates and codify data engineering best practices.
Requirements
- Proven ownership of production data pipelines at significant scale.
- Strong production implementation skills in Java and/or Python plus SQL.
- Experience making batch-versus-streaming decisions and adding near-real-time updates such as CDC or event streams.
- Experience with Kafka and familiarity with Flink, Flink SQL, and streaming fundamentals.
- A strong data-correctness mindset and robust observability experience.
- A track record of improving pipeline availability, freshness, and correctness.
- Experience setting technical direction across multiple pipelines or domains.
- Search-indexing exposure and experience with document or feature pipelines are bonus qualifications.
- Familiarity with Spark, MapReduce, BigQuery, GCP, Iceberg, Hudi, Kafka, or Flink is a bonus.
- Deeper Flink knowledge, including state, checkpointing, and savepoints, is a bonus.
Benefits
- Remote-first, digital-by-default work arrangement with distributed teams across North America and beyond.
Tech Stack
Categories
Data Engineering
About Shopify
Shopify is a leading global commerce company, providing trusted tools to start, grow, market, and manage a retail business of any size. Shopify makes commerce better for everyone with a platform and services that are engineered for reliability, while delivering a better shopping experience for consumers everywhere. Shopify powers millions of businesses in more than 175 countries and is trusted by brands such as Allbirds, Gymshark, PepsiCo, Staples, and many more. Find all our jobs here: www.shopify.com/careers