4 months ago
Toronto, CanadaSenior
Responsibilities
- Lead the architecture, design, implementation, and production delivery of enterprise-scale data platforms across multiple client engagements.
- Define solution architectures, technical roadmaps, implementation strategies, and engineering standards.
- Conduct architecture and code reviews, mentor Data Engineers, and lead complex technical initiatives.
- Partner with enterprise clients through discovery workshops, architecture sessions, technical planning, solution presentations, and delivery.
- Design and optimize Databricks data pipelines, ETL/ELT frameworks, Lakehouse architectures, Delta Lake implementations, and Apache Spark applications.
- Integrate structured and unstructured data from enterprise applications, APIs, streaming platforms, and cloud storage.
- Architect cloud-native platforms across AWS, Azure, or Google Cloud Platform and implement Infrastructure-as-Code and CI/CD pipelines.
- Implement data governance, security, metadata management, lineage, role-based access controls, data quality, monitoring, and observability.
- Enable AI and machine learning initiatives through scalable feature engineering pipelines and production-ready datasets.
Requirements
- Bachelor's or master's degree in Computer Science, Data Engineering, Software Engineering, or a related technical discipline.
- 8+ years of experience designing and building enterprise-scale data platforms.
- 5+ years of hands-on experience with Databricks and Apache Spark.
- Strong expertise in Python, SQL, Spark, Delta Lake, Lakehouse architecture, distributed computing, data modeling, and ETL/ELT frameworks.
- Experience with AWS, Azure, or Google Cloud Platform, CI/CD pipelines, and Infrastructure-as-Code.
- Experience with data governance, security, metadata management, data quality, and performance and cost optimization.
- Excellent communication and stakeholder management skills, including direct work with enterprise clients.
- Demonstrated ability to mentor engineers and lead technical initiatives.
- Preferred qualifications include Databricks Certified Professional Data Engineer certification; experience with Delta Live Tables, MLflow, Unity Catalog, Databricks SQL, Kafka, Kinesis, Event Hubs, Snowflake, dbt, Airflow, Kubernetes, Docker, Terraform, AI/ML data platforms, Feature Stores, Retrieval-Augmented Generation architectures, and consulting or professional services.
Tech Stack
Apache AirflowApache KafkaApache SparkAWSAzureDatabricksdbtDockerGoogle Cloud PlatformKubernetesMLflowPythonSnowflakeSQLTerraform
Categories
Data Engineering
About ShyftLabs
ShyftLabs builds and runs enterprise data and AI platforms, combining data engineering, analytics, and responsible AI controls with consulting and managed delivery. It designs governed data infrastructure, customer and retail analytics, and omnichannel marketing data products for large organizations. Founded in 2018 and headquartered in Toronto, the privately held firm serves global clients and staffs teams across Toronto, New York, Dubai, and New Delhi.