Data Engineer
Sigma Software8 days ago
Remote, Poland or Warsaw, PolandMid Level
Responsibilities
- Design, develop, and maintain scalable data platforms and pipelines for analytics and operational use cases.
- Build distributed data processing workflows using Apache Spark, PySpark, and Databricks.
- Develop cloud-native data solutions using Azure and AWS services.
- Model and optimize data structures for analytics, reporting, and downstream consumption.
- Integrate heterogeneous data sources while ensuring data quality, consistency, and traceability.
- Implement data quality controls, schema validation, lineage tracking, and monitoring processes.
- Optimize pipeline and infrastructure performance, scalability, reliability, and cloud costs.
- Collaborate with scientists, analysts, and engineers to translate business and research requirements into technical solutions.
- Contribute to infrastructure automation, CI/CD processes, automated testing, and engineering best practices.
- Research and introduce new technologies and approaches in data engineering.
- Support scientific experimentation, analytics, machine learning workflows, governance, and reproducibility in a fusion energy research project.
Requirements
- At least 3 years of commercial Data Engineering experience.
- At least 2 years of hands-on production experience with Databricks.
- Strong expertise with Delta Lake, including medallion architecture and table optimization.
- Advanced proficiency in Python, PySpark, and SQL.
- Deep practical knowledge of Databricks components including Unity Catalog, Lakeflow Jobs/Workflows, and job optimization.
- Hands-on AWS experience with S3, IAM, KMS, and at least one of AWS Batch, ECR, EventBridge, Lambda, or API Gateway.
- Experience designing event-driven and asynchronous data pipelines.
- Strong knowledge of Infrastructure as Code and CI/CD practices, with practical Terraform and Databricks Asset Bundles experience.
- Experience with Git workflows, pull request reviews, and automated testing.
- Understanding of secure authentication approaches including OIDC and service principals.
- Practical experience with MLflow, experiment tracking, model registry, and reproducibility practices.
- Strong understanding of data quality, lineage, schema contracts, and provenance.
- Experience with Great Expectations, Soda, pyDeequ, or Databricks DQX.
- Experience monitoring and optimizing data platform reliability and cloud costs.
- Upper-Intermediate or higher English is a plus.
- Docker and AWS batch workload experience is a plus.
- Familiarity with ECR and containerized data processing solutions is a plus.
- Experience with scikit-optimize, BoTorch, Ax, or scipy is a plus.
- Understanding of ML feature engineering and leakage prevention is a plus.
- Experience in regulated or high-security environments and knowledge of data governance, compliance, and data residency are a plus.
- Experience disabling or restricting AI-assisted tooling and features in secure environments is a plus.
Categories
Data Engineering
About Sigma Software
Sigma Software is a London-headquartered, privately held IT consultancy that designs and builds custom software for startups and enterprises. The company provides end-to-end services—engineering, web and mobile development, .NET/Java platforms, AR/VR, and UI/UX—delivered via outsourcing, managed teams, and consulting. Founded in 2002, it supports long-term product development and modernization projects across multiple industries for global clients.