Senior Data Engineer
Sigma Software15 days ago
Remote, Poland or Warsaw, PolandSenior
Responsibilities
- Design and build scalable cloud-native data platforms from greenfield development through production.
- Design and implement near-real-time ingestion pipelines using event-driven patterns.
- Define and enforce Data Lake/Lakehouse standards, medallion architecture, and data contracts.
- Refactor, optimize, and modernize Spark and PySpark scripts.
- Establish best practices for code quality, testing, and CI/CD across data pipelines.
- Drive adoption of AI tooling and agentic workflows within the data engineering team.
- Ensure data quality, observability, reliability, lineage, provenance, schema contracts, and versioning across pipelines and platforms.
- Design and deliver self-service tooling and microservices that abstract platform complexity.
- Collaborate directly with the customer’s scientific and engineering teams.
- Monitor platform performance, reliability, and cloud infrastructure costs.
Requirements
- 5+ years of experience in data engineering.
- 3+ years of production experience with Databricks and Delta Lake.
- Strong expertise in medallion architecture and event-driven asynchronous pipeline design.
- Advanced proficiency in Python, PySpark, and SQL.
- Advanced hands-on experience with Databricks, including Unity Catalog, Lakeflow Jobs/Workflows, and job optimization.
- Strong AWS expertise, including S3, IAM, KMS, AWS Batch, ECR, EventBridge, Lambda, or API Gateway.
- Experience with infrastructure as code and CI/CD using Terraform and Databricks Asset Bundles.
- Practical experience with Git workflows, pull request reviews, and automated testing.
- Experience configuring secure CI pipelines using OIDC or service principals.
- Practical MLflow experience with experiment tracking, model registration, and reproducibility practices.
- Experience with data quality frameworks such as Great Expectations, Soda, pyDeequ, or Databricks Labs DQX.
- Experience with monitoring, observability, and cloud cost optimization.
- Upper-Intermediate English level or higher.
- Preferred experience packaging workloads in Docker containers and running batch jobs on AWS.
- Preferred familiarity with scikit-optimize, BoTorch, Ax, and scipy.
- Preferred experience building feature sets and train/test splits that avoid data leakage.
- Preferred experience in regulated or high-security environments with data residency requirements.
- Preferred experience disabling or restricting AI features in secured environments.
- Strong analytical mindset, ownership, accountability, communication, proactive problem-solving, scalability focus, maintainability focus, and collaborative teamwork.
Benefits
- Opportunity to work on an ambitious deep-tech initiative supporting the development of the world’s first commercial laser fusion power plant.
- Opportunity to contribute to breakthrough research in science, clean energy, automation, and AI-enabled data engineering.
- Global project opportunities through Sigma Software.
About Sigma Software
Sigma Software is a London-headquartered, privately held IT consultancy that designs and builds custom software for startups and enterprises. The company provides end-to-end services—engineering, web and mobile development, .NET/Java platforms, AR/VR, and UI/UX—delivered via outsourcing, managed teams, and consulting. Founded in 2002, it supports long-term product development and modernization projects across multiple industries for global clients.