Sr. Data Engineer
Bits In Glass2 months ago
Remote, United States +3 moreSenior
Responsibilities
- Design and implement scalable data lakes, data warehouses, analytics platforms, and end-to-end data architectures.
- Define data integration, transformation, governance, security, and privacy strategies across cloud platforms.
- Develop data models, ETL/ELT pipelines, ingestion frameworks, and structured and unstructured data solutions.
- Build and optimize large-scale batch and streaming pipelines using Databricks, Spark, Kafka, and related technologies.
- Provide technical leadership and mentorship to data engineers and developers.
- Optimize data architectures and processes for performance, reliability, scalability, and compliance.
- Lead data migration projects from traditional warehouses to the Databricks Lakehouse.
- Advise clients on data modernization, architecture decisions, and engineering best practices.
- Collaborate with stakeholders, data engineers, and analytics teams to translate business requirements into technical solutions.
Requirements
- At least 5 years of experience in data architecture, data engineering, or analytics solution design.
- Hands-on experience with Databricks, Snowflake, Redshift, or Synapse data lake and warehouse technologies.
- Deep knowledge of data modeling, data integration, and ETL/ELT design.
- Proficiency in SQL and one or more of PySpark, Python, or Scala for complex transformations and Spark optimization.
- Proven experience designing and optimizing large-scale ingestion pipelines using Databricks Autoloader.
- Practical experience building reliable, self-managing ETL/ELT pipelines with Delta Live Tables.
- Experience building high-throughput, low-latency streaming ingestion solutions with Apache Kafka, Spark Structured Streaming, and Databricks Streaming.
- Extensive experience applying Medallion architecture within Databricks environments.
- Experience designing CI/CD pipelines for Databricks workflows, notebooks, and cluster configurations using Azure DevOps, GitHub Actions, or GitLab CI.
- Working knowledge of AWS or Azure data storage, networking, and security concepts relevant to Databricks deployments.
- Ability to engage clients, present technical solutions, and communicate complex ideas clearly.
- Bachelor’s or master’s degree in Computer Science, Information Systems, Engineering, or a related field.
- Experience with AI/ML integration, data science workflows, data cataloging, metadata management, consulting, or client-facing technology services is preferred.
Tech Stack
Amazon RedshiftApache KafkaApache SparkAWSAzureDatabricksGitHub ActionsGitLab CI/CDPythonScalaSnowflakeSQL
Categories
Data Engineering
About Bits In Glass
Bits In Glass is an IT consulting firm that designs and implements enterprise automation, integration, and CRM solutions using platforms like Appian, MuleSoft, Salesforce, Pega, and Blue Prism. It delivers project-based and managed services to large organizations across North America, Europe, and India, covering AI, cloud, data, and business applications. Founded in 2002 and headquartered in Edmonton, it is privately held and received private‑equity investment in 2024.