Senior Data Engineer (Databricks Migration)
Sigma Software3 months ago
Remote, Portugal or Lisbon, PortugalSenior
Responsibilities
- Migrate a large-scale analytical platform from BigQuery to Databricks.
- Design and implement scalable Databricks Lakehouse architectures using Delta Lake.
- Analyze existing ETL/ELT workloads and define migration approaches.
- Develop and optimize large-scale data pipelines using Apache Spark and PySpark.
- Implement incremental processing strategies and scalable transformation frameworks.
- Design and maintain Bronze, Silver, and Gold medallion architecture layers.
- Implement data governance and security best practices using Unity Catalog.
- Develop reusable data platform components and engineering standards.
- Participate in architecture discussions and technical solution design.
- Conduct code reviews and contribute to platform reliability and maintainability.
- Troubleshoot and optimize complex SQL and Spark workloads.
- Support production deployments and platform modernization activities.
- Collaborate with international, cross-functional Data Science, Analytics, Product, and Customer Engineering teams.
Requirements
- At least 5 years of professional experience as a Data Engineer.
- Strong programming skills in Python and advanced SQL.
- Hands-on commercial experience with Databricks, Apache Spark, PySpark, and Delta Lake.
- Experience designing and building modern cloud-based data platforms.
- Experience developing ETL/ELT pipelines and large-scale data processing solutions.
- Experience with Spark Declarative Pipelines, cluster monitoring, metrics analysis, and performance optimization.
- Strong understanding of distributed data processing architectures, data warehousing, and dimensional modeling.
- Experience with Airflow or similar orchestration tools.
- Experience optimizing complex analytical SQL workloads and implementing CI/CD practices for data engineering platforms.
- Strong troubleshooting and performance optimization skills.
- Ability to work independently and collaboratively in cross-functional international teams.
- Upper-Intermediate or higher English level.
- Preferred experience with GCP, AWS, or Azure cloud platforms.
- Preferred experience in retail analytics, pricing optimization, machine learning or AI-related data workloads, platform modernization, or cloud migration initiatives.
Benefits
- Opportunities for continuous learning and technology growth.
- Meaningful engineering impact on complex international projects.
- Collaboration with international teams and exposure to enterprise-scale retail analytics modernization.
Tech Stack
Categories
Data Engineering
About Sigma Software
Sigma Software is a London-headquartered, privately held IT consultancy that designs and builds custom software for startups and enterprises. The company provides end-to-end services—engineering, web and mobile development, .NET/Java platforms, AR/VR, and UI/UX—delivered via outsourcing, managed teams, and consulting. Founded in 2002, it supports long-term product development and modernization projects across multiple industries for global clients.