Senior Data Engineer
Datavail, LLC10 days ago
Remote, United StatesSenior
Responsibilities
- Design, build, and maintain production-grade batch, incremental, real-time, and near-real-time data pipelines across Microsoft Fabric and Azure.
- Implement CDC, watermarking, merge/upsert, late-arriving-data, and idempotent reprocessing patterns.
- Develop PySpark, Spark SQL, and T-SQL transformations across bronze, silver, and gold Delta Lake layers.
- Build Fabric solutions using OneLake, Lakehouse, Warehouse, Data Factory, Dataflows Gen2, Spark notebooks, Real-Time Intelligence, and Power BI.
- Implement dimensional and Data Vault models, data-quality testing, reconciliation, observability, logging, alerting, and SLA monitoring.
- Establish Git-based CI/CD and automated deployments using Azure DevOps or GitHub, Fabric deployment pipelines, REST APIs, Bicep, and Terraform.
- Apply Purview catalog and lineage, least-privilege access, sensitivity controls, encryption, and secure connectivity.
- Support migrations from SQL Server, SSIS, SSAS, legacy warehouses, and Synapse to Fabric and Azure, including conversion, reconciliation, and cutover.
- Prepare governed data for AI/ML and RAG workloads, including ingestion, chunking, embeddings, indexing, and telemetry.
- Lead engineering workstreams, estimates, risk management, client communication, mentoring, reusable accelerators, and operational handover.
Requirements
- 8+ years of experience in data engineering or data warehousing.
- 3+ years delivering Azure solutions and production Microsoft Fabric experience.
- Advanced T-SQL, Spark SQL, Python, and PySpark skills.
- Working knowledge of KQL and DAX.
- Deep experience with Delta Lake, Spark tuning, medallion/lakehouse patterns, CDC, and incremental loading.
- Strong dimensional modeling experience and working knowledge of Data Vault 2.0.
- Experience with CI/CD, Git, automated testing, infrastructure as code, DataOps, governance, and security.
- Professional-services or client-facing delivery experience.
- Authorization to work in the United States and ability to travel up to 15%.
- Preferred qualifications include DP-700, DP-600, or DP-203 certification; streaming/event-driven systems, RAG/GenAI, Fabric IQ/data agents, and relevant industry experience.
Benefits
- The role requires authorization to work in the United States and travel of up to 15%.
Tech Stack
Categories
Data Engineering
About Datavail, LLC
Datavail is an IT services consultancy and managed services provider that helps enterprises run and modernize data platforms, analytics, cloud, and enterprise applications. It delivers database administration and data management across Oracle, Microsoft SQL Server, MySQL, and MongoDB, plus BI/DW, analytics, and application managed services. Founded in 2007 and headquartered in Boulder, Colorado, the company is privately held and backed by private-equity owner CIVC Partners.