Senior Data Engineer (Azure Databricks)
Applaudo Studios20 days ago
Remote, Costa Rica or San José, Costa RicaSenior
Responsibilities
- Design, build, and maintain reliable production-grade data pipelines using Azure Databricks and Python.
- Develop scalable data transformations with PySpark or Spark SQL.
- Ingest and process data from APIs, relational databases, cloud storage, and SFTP sources.
- Integrate data from ticketing, CRM, marketing, retail, and other enterprise platforms into a centralized Data Hub.
- Develop and maintain data warehouse structures, dimensional models, and data marts.
- Write, optimize, and maintain SQL transformations for data accuracy and processing efficiency.
- Implement data quality validations and consistency checks between source systems and target models.
- Build and maintain outbound data feeds for third-party systems.
- Investigate data discrepancies, troubleshoot pipeline issues, and implement corrective actions.
- Document pipelines, transformation logic, dependencies, and technical processes.
- Collaborate with stakeholders to clarify requirements, identify risks, and deliver reliable data solutions.
- Communicate progress, blockers, and potential delays proactively.
Requirements
- 5+ years of professional data engineering experience designing, building, and maintaining production-grade data pipelines.
- Strong hands-on experience with Azure Databricks.
- Advanced Python skills for data processing, transformation, and pipeline development.
- Solid Apache Spark experience, including PySpark and/or Spark SQL.
- Advanced SQL proficiency, including complex transformations, data validation, and query optimization.
- Strong knowledge of data warehousing and dimensional modeling, including facts, dimensions, and data marts.
- Experience integrating data from REST APIs, relational databases, cloud storage, and SFTP.
- Hands-on experience with Microsoft Azure data services for ingestion, storage, processing, and integration.
- Experience implementing data quality checks and validating consistency between source systems and target data models.
- Ability to troubleshoot pipeline failures and investigate data discrepancies.
- Strong problem-solving, analytical, organizational, ownership, autonomy, documentation, and communication skills.
- Strong English communication skills and ability to collaborate with a US-based team.
- Experience migrating SQL-based data workloads to Azure Databricks is preferred.
Tech Stack
Categories
Data Engineering
About Applaudo Studios
Applaudo Studios provides nearshore software engineering and AI consulting to enterprises, delivering product design, custom development, cloud, and data services from Latin America. Founded in 2013 and headquartered in San Salvador, the privately held firm builds and operates web, mobile, and enterprise systems and supports digital transformation initiatives. Clients engage Applaudo for AI assessments and implementation, scalable delivery teams, and ongoing DevOps and QA support.