Senior Data Engineer
Sigma Software15 days ago
Remote, Poland or Warsaw, PolandSenior
Responsibilities
- Design and build scalable cloud-native data platforms from greenfield development through production.
- Implement near-real-time ingestion pipelines using event-driven and asynchronous patterns.
- Define and enforce data lake, lakehouse, medallion architecture, and data contract standards.
- Refactor, optimize, and modernize Spark and PySpark scripts and data pipelines.
- Establish best practices for code quality, testing, CI/CD, data quality, observability, and reliability.
- Drive adoption of AI tooling and agentic workflows within the data engineering team.
- Design and deliver self-service tooling and microservices that abstract platform complexity.
- Collaborate directly with the customer’s scientific and engineering teams.
- Monitor platform performance, reliability, and cloud infrastructure costs.
Requirements
- 5+ years of experience in data engineering.
- 3+ years of production experience with Databricks and Delta Lake.
- Strong expertise in medallion architecture and event-driven asynchronous pipeline design.
- Advanced proficiency in Python, PySpark, and SQL.
- Hands-on experience with Databricks Unity Catalog, Lakeflow Jobs/Workflows, and job optimization.
- Strong AWS experience, including S3, IAM, KMS, AWS Batch, ECR, EventBridge, Lambda, or API Gateway.
- Experience with Terraform, Databricks Asset Bundles, infrastructure as code, and CI/CD.
- Practical experience with Git workflows, pull request reviews, and automated testing.
- Experience configuring secure CI pipelines using OIDC or service principals.
- Practical MLflow experience with experiment tracking, model registration, and reproducibility.
- Understanding of data quality, lineage, provenance, schema contracts, and versioning.
- Experience with data quality frameworks such as Great Expectations, Soda, pyDeequ, or Databricks Labs DQX.
- Experience with monitoring, observability, and cloud cost optimization.
- Upper-Intermediate English level or higher.
- Preferred experience with Docker, AWS batch jobs, scikit-optimize, BoTorch, Ax, scipy, feature sets, leakage-resistant train/test splits, regulated or high-security environments, data residency requirements, and restricting AI features in secured environments.
- Strong analytical, communication, ownership, problem-solving, scalability, maintainability, and collaboration skills.
About Sigma Software
Sigma Software is a London-headquartered, privately held IT consultancy that designs and builds custom software for startups and enterprises. The company provides end-to-end services—engineering, web and mobile development, .NET/Java platforms, AR/VR, and UI/UX—delivered via outsourcing, managed teams, and consulting. Founded in 2002, it supports long-term product development and modernization projects across multiple industries for global clients.