Data Engineer (DATABRICKS / PYSPARK)
Sigma Software9 days ago
Remote, Poland or Warsaw, PolandMid Level
Responsibilities
- Develop and maintain batch and near real-time data pipelines for large volumes of data.
- Transform, validate, optimize, and monitor data pipelines using Databricks, PySpark, and SQL.
- Build, test, and automate ETL processes and data workflows.
- Support scalable Data Lakehouse solutions and Medallion Architecture implementations.
- Improve data quality, organization, and reliability across platforms.
- Troubleshoot pipeline issues and optimize high-load distributed data processing systems.
- Collaborate with engineers, analysts, and stakeholders to deliver scalable solutions.
- Ensure deliverables meet quality standards and project requirements.
- Stay current with modern data engineering technologies.
Requirements
- At least 2 years of experience in Big Data development and database design.
- Strong hands-on SQL experience.
- Strong Python development skills.
- Hands-on experience with PySpark.
- Experience developing and maintaining data pipelines in Databricks.
- Experience with Delta Lake and Medallion Architecture using Bronze, Silver, and Gold layers.
- Strong analytical and problem-solving skills.
- Intermediate or higher English level is a plus.
- Experience with AWS services, including S3, is a plus.
About Sigma Software
Sigma Software is a London-headquartered, privately held IT consultancy that designs and builds custom software for startups and enterprises. The company provides end-to-end services—engineering, web and mobile development, .NET/Java platforms, AR/VR, and UI/UX—delivered via outsourcing, managed teams, and consulting. Founded in 2002, it supports long-term product development and modernization projects across multiple industries for global clients.