8 months ago
Calgary, CanadaSenior
Responsibilities
- Design, build, and maintain scalable batch and real-time ETL/ELT pipelines using GCP Dataflow, Cloud Functions, Pub/Sub, and Cloud Composer.
- Architect and implement high-volume data ingestion, processing, and warehouse infrastructure.
- Develop and manage the central data warehouse in Google BigQuery, including data models, schemas, table structures, partitioning, clustering, query optimization, and materialized views.
- Write SQL and Python code to transform raw data into curated, analysis-ready datasets.
- Build transformation workflows and curated data models supporting analytics, reporting, BI, and data science initiatives.
- Monitor, troubleshoot, and optimize data infrastructure for performance, reliability, and cost efficiency.
- Implement automated data quality checks, validation rules, monitoring frameworks, governance processes, observability, and lineage tracking.
- Ensure data is accessible for Looker and other analytics platforms.
- Collaborate with software engineers, data analysts, data scientists, product teams, leadership, clients, and stakeholders to deliver scalable data solutions.
- Own data platform and architecture decisions and identify opportunities to improve reliability, automate workflows, and generate insights.
Requirements
- At least 5 years of hands-on experience in data engineering, data integration, or data platform development.
- Degree in Computer Science, Engineering, Mathematics, or a related STEM discipline.
- Strong programming and query skills in SQL and Python.
- Experience with Git and Agile/Scrum environments.
- Experience designing and orchestrating ETL pipelines, particularly with Databricks.
- Experience working in cloud environments such as GCP, AWS, or Azure.
- Experience with MongoDB and Elasticsearch.
- Strong understanding of data warehousing and dimensional modeling methodologies.
- Hands-on experience with Airflow and Hadoop.
- Experience using Docker for containerized workflows and reproducible environments.
- Ability to improve data quality, reliability, and automation, with strong business awareness and communication skills.
- Retail industry experience is a plus.
- A master’s degree in Computer Science, Engineering, or a related discipline is preferred.
- Experience with enterprise-scale data platforms and Fortune 500 clients is preferred.
- Familiarity with Druid, its Python API, and Kafka integrations is preferred.
- Strong Apache Spark experience and experience designing real-time streaming data architectures are preferred.
- Experience with AI-driven platforms, AI/ML data infrastructure, or agentic AI workflows is preferred.
Tech Stack
Apache AirflowApache HadoopApache KafkaApache SparkAWSAzureDatabricksDockerElasticsearchGitGoogle BigQueryGoogle Cloud PlatformMongoDBPythonSQL
Categories
Data Engineering
About ShyftLabs
ShyftLabs builds and runs enterprise data and AI platforms, combining data engineering, analytics, and responsible AI controls with consulting and managed delivery. It designs governed data infrastructure, customer and retail analytics, and omnichannel marketing data products for large organizations. Founded in 2018 and headquartered in Toronto, the privately held firm serves global clients and staffs teams across Toronto, New York, Dubai, and New Delhi.