Sr. Data Engineer
Bits In Glass2 months ago
Remote, WorldwideSenior
Responsibilities
- Design and implement end-to-end data architectures, including data lakes, data warehouses, and analytics platforms.
- Define scalable, secure, and high-performance data integration and transformation strategies.
- Translate business requirements into technical solutions supporting analytics, reporting, and AI initiatives.
- Develop data models, ETL/ELT pipelines, and frameworks for structured and unstructured data.
- Design and optimize large-scale ingestion pipelines using Databricks Autoloader.
- Build and manage reliable self-managing ETL/ELT pipelines using Delta Live Tables.
- Develop high-throughput, low-latency streaming ingestion solutions using Apache Kafka, Spark Structured Streaming, and Databricks Streaming.
- Apply Medallion architecture across Bronze, Silver, and Gold layers in Databricks environments.
- Design CI/CD pipelines for Databricks workflows, notebooks, and cluster configurations.
- Plan and execute migrations from traditional data warehouses to the Databricks Lakehouse.
- Provide technical leadership and mentorship to data engineers and developers.
- Ensure data governance, security, privacy, performance, and reliability across platforms.
- Advise clients on architecture decisions, data modernization, and best practices.
- Collaborate with stakeholders, data engineers, and analytics teams and communicate complex technical solutions clearly.
Requirements
- At least 5 years of experience in data architecture, data engineering, or analytics solution design.
- Hands-on experience with Databricks, Snowflake, Redshift, or Synapse data lake and warehouse technologies.
- Deep understanding of data modeling, data integration, and ETL/ELT design.
- Proficiency in SQL and one or more of PySpark, Python, or Scala, particularly for complex transformations and Spark optimization.
- Experience with Databricks Autoloader, Delta Live Tables, Apache Kafka, Spark Structured Streaming, and Databricks Streaming.
- Extensive experience applying Medallion architecture in Databricks environments.
- Experience designing CI/CD pipelines using Azure DevOps, GitHub Actions, or GitLab CI for Databricks deployments and testing.
- Strong knowledge of AWS or Azure data storage, networking, and security concepts relevant to Databricks.
- Bachelor’s or master’s degree in Computer Science, Information Systems, Engineering, or a related field.
- Proven ability to engage clients, present technical solutions, and explain complex ideas clearly.
- Strong problem-solving, communication, and collaboration skills.
- Preferred qualifications include AI/ML integration, data science workflows, data cataloging, metadata management, and technology consulting experience.
Tech Stack
Amazon RedshiftApache KafkaApache SparkAWSAzureDatabricksGitHub ActionsGitLab CI/CDPythonScalaSnowflakeSQL
Categories
Data Engineering
About Bits In Glass
Bits In Glass is an IT consulting firm that designs and implements enterprise automation, integration, and CRM solutions using platforms like Appian, MuleSoft, Salesforce, Pega, and Blue Prism. It delivers project-based and managed services to large organizations across North America, Europe, and India, covering AI, cloud, data, and business applications. Founded in 2002 and headquartered in Edmonton, it is privately held and received private‑equity investment in 2024.