Sr. Data Engineer
Bits In Glass2 months ago
Remote, United States +3 moreSenior
Responsibilities
- Design and implement end-to-end data architectures including data lakes, data warehouses, and analytics platforms.
- Define scalable data integration, transformation, governance, security, and privacy strategies.
- Translate business requirements into technical data solutions supporting analytics, reporting, and AI initiatives.
- Develop data models, ETL/ELT pipelines, and frameworks for structured and unstructured data.
- Build large-scale ingestion pipelines using Databricks Autoloader and reliable self-managing pipelines using Delta Live Tables.
- Design high-throughput, low-latency streaming ingestion solutions using Apache Kafka, Spark Structured Streaming, and Databricks Streaming.
- Apply Medallion architecture across Bronze, Silver, and Gold layers in Databricks environments.
- Design CI/CD pipelines for Databricks workflows, notebooks, and cluster configurations.
- Plan and execute migrations from traditional data warehouses to the Databricks Lakehouse.
- Optimize data architectures and processes for performance and reliability.
- Provide technical leadership and mentorship to data engineers and developers.
- Advise clients on data architecture, modernization, and technical best practices.
- Collaborate with business stakeholders, data engineers, and analytics teams.
- Stay current with cloud data services and emerging technologies.
Requirements
- 5+ years of experience in data architecture, data engineering, or analytics solution design.
- Hands-on experience with Databricks, Snowflake, Redshift, or Synapse data lake and warehouse technologies.
- Deep understanding of data modeling, data integration, and ETL/ELT design.
- Proficiency in SQL and one or more programming languages, including PySpark/Python or Scala, for complex transformations and Spark optimization.
- Experience designing, implementing, and optimizing large-scale Databricks Autoloader ingestion pipelines.
- Experience building and managing Delta Live Tables ETL/ELT pipelines.
- Experience with Apache Kafka, Spark Structured Streaming, and Databricks Streaming.
- Extensive experience applying Medallion architecture in Databricks.
- Experience with Azure DevOps, GitHub Actions, or GitLab CI for Databricks CI/CD workflows.
- Experience planning and executing data warehouse migrations to the Databricks Lakehouse.
- Working knowledge of AWS or Azure data storage, networking, and security concepts relevant to Databricks.
- Ability to engage clients, present technical solutions, and communicate complex ideas clearly.
- Bachelor’s or master’s degree in Computer Science, Information Systems, Engineering, or a related field.
- Experience with AI/ML integration and data science workflows is preferred.
- Knowledge of data cataloging and metadata management tools is preferred.
- Prior consulting or client-facing experience in a technology services firm is preferred.
- Strong problem-solving, communication, and collaboration skills.
Benefits
- Supportive, collaborative, and people-focused team environment.
- Opportunities to work on meaningful real-world business challenges and deepen technical expertise.
- Global recognition as a Great Place to Work and multiple industry partner awards.
- Client-facing consulting experience across Canada, the UK, India, and the US.
- Role based within a global organization with offices in Edmonton, Calgary, Toronto, Denver, India, and the United Kingdom.
Tech Stack
Amazon RedshiftApache KafkaApache SparkAWSAzureDatabricksGitHub ActionsGitLab CI/CDPythonScalaSnowflakeSQL
Categories
Data Engineering
About Bits In Glass
Bits In Glass is an IT consulting firm that designs and implements enterprise automation, integration, and CRM solutions using platforms like Appian, MuleSoft, Salesforce, Pega, and Blue Prism. It delivers project-based and managed services to large organizations across North America, Europe, and India, covering AI, cloud, data, and business applications. Founded in 2002 and headquartered in Edmonton, it is privately held and received private‑equity investment in 2024.