Data Engineer
Sigma Software14 days ago
Remote, EMEA or Warsaw, PolandMid Level
Responsibilities
- Design, develop, and maintain scalable data platforms and pipelines for analytics and operational use cases.
- Build distributed data processing workflows using Apache Spark, PySpark, and Databricks.
- Develop cloud-native data solutions leveraging AWS and Azure services.
- Model and optimize data structures for analytics, reporting, and downstream consumption.
- Integrate heterogeneous data sources while ensuring data quality, consistency, traceability, schema validation, lineage, and monitoring.
- Optimize pipeline and infrastructure performance, scalability, reliability, and cloud cost efficiency.
- Collaborate with scientists, analysts, and engineering teams to translate business and research requirements into technical solutions.
- Contribute to CI/CD, infrastructure automation, Git workflows, pull request reviews, automated testing, and engineering best practices.
- Research and introduce technologies and approaches within the data engineering ecosystem.
- Support experimentation, analytics, machine learning, and operational workflows for a fusion energy research project.
Requirements
- At least 3 years of commercial Data Engineering experience.
- At least 2 years of hands-on production experience with Databricks.
- Strong expertise with Delta Lake, medallion architecture, and table optimization.
- Advanced proficiency in Python, PySpark, and SQL.
- Deep practical knowledge of Databricks components including Unity Catalog, Lakeflow Jobs/Workflows, and job optimization.
- Hands-on AWS experience including S3, IAM, KMS, and at least one of AWS Batch, ECR, EventBridge, Lambda, or API Gateway.
- Experience designing event-driven and asynchronous data pipelines.
- Strong knowledge of Infrastructure as Code and CI/CD practices.
- Practical experience with Terraform and Databricks Asset Bundles.
- Understanding of secure authentication approaches such as OIDC and service principals.
- Practical experience with MLflow, experiment tracking, model registry, and reproducibility practices.
- Strong understanding of data quality, lineage, schema contracts, and provenance.
- Experience with data quality frameworks such as Great Expectations, Soda, pyDeequ, or Databricks DQX.
- Upper-Intermediate or higher English is a plus.
- Experience packaging workloads with Docker and running batch workloads in AWS is a plus.
- Experience with ECR and containerized data processing solutions is a plus.
- Familiarity with scikit-optimize, BoTorch, Ax, or SciPy is a plus.
- Understanding of ML feature engineering and leakage prevention is a plus.
- Experience in regulated or high-security environments is a plus.
- Knowledge of data governance, compliance, and data residency requirements is a plus.
- Experience restricting AI-assisted tooling or features in secure environments is a plus.
Categories
Data Engineering
About Sigma Software
Sigma Software is a London-headquartered, privately held IT consultancy that designs and builds custom software for startups and enterprises. The company provides end-to-end services—engineering, web and mobile development, .NET/Java platforms, AR/VR, and UI/UX—delivered via outsourcing, managed teams, and consulting. Founded in 2002, it supports long-term product development and modernization projects across multiple industries for global clients.