Senior Data Engineer (Databricks Migration)
Sigma Software3 months ago
Remote, Poland or Warsaw, PolandSenior
Responsibilities
- Migrate analytical platform workloads from BigQuery to Databricks.
- Design and implement scalable Lakehouse architectures using Databricks and Delta Lake.
- Analyze existing ETL/ELT workloads and define migration approaches.
- Develop and optimize high-volume data pipelines and Spark-based processing solutions with PySpark.
- Implement incremental processing strategies, reusable transformation frameworks, and Bronze, Silver, and Gold medallion layers.
- Implement data governance and security best practices with Unity Catalog.
- Design reusable data platform components and engineering standards.
- Analyze and optimize complex SQL and Spark workloads, including cluster monitoring, metrics, and performance.
- Participate in architecture discussions, technical solution design, code reviews, deployments, and platform modernization.
- Collaborate with Data Science, Analytics, Product, and Customer Engineering teams.
- Troubleshoot production issues and contribute to platform reliability and maintainability.
Requirements
- At least 5 years of professional experience as a Data Engineer.
- Strong Python programming skills and advanced SQL proficiency.
- Hands-on commercial experience with Databricks, Apache Spark, PySpark, and Delta Lake.
- Experience designing and building modern cloud-based data platforms.
- Experience developing ETL/ELT pipelines and large-scale data processing solutions.
- Experience with Spark Declarative Pipelines, cluster monitoring, metrics analysis, and performance optimization.
- Strong understanding of distributed data processing architectures, data warehousing, and dimensional modeling.
- Experience with Airflow or similar orchestration tools.
- Experience optimizing complex analytical SQL workloads and implementing CI/CD practices for data engineering platforms.
- Strong troubleshooting, analytical, problem-solving, communication, and collaboration skills.
- Upper-Intermediate or higher English proficiency.
- Preferred: experience with GCP, AWS, or Azure cloud platforms.
- Preferred: experience in retail analytics, pricing optimization, machine learning or AI-related data workloads, platform modernization, or cloud migration initiatives.
Benefits
- Opportunities for continuous learning and technology growth.
- Work on complex international projects with cross-functional teams.
- Opportunity to contribute to meaningful modernization of an enterprise-scale analytics ecosystem.
Tech Stack
Categories
Data Engineering
About Sigma Software
Sigma Software is a London-headquartered, privately held IT consultancy that designs and builds custom software for startups and enterprises. The company provides end-to-end services—engineering, web and mobile development, .NET/Java platforms, AR/VR, and UI/UX—delivered via outsourcing, managed teams, and consulting. Founded in 2002, it supports long-term product development and modernization projects across multiple industries for global clients.