Senior Data Engineer
Sigma Software3 days ago
Remote, Poland +2 moreSenior
Responsibilities
- Write and defend diagnostic SQL queries against large-scale production datasets.
- Build and maintain ingestion pipelines for bid, win, and impression logs in BigQuery.
- Harmonize independently designed datasets and maintain versioned field mappings.
- Develop point-in-time-correct feature tables, aggregation pipelines, conversion pipelines, and labeling pipelines with delayed-label handling.
- Own the data-serving write path, schema contracts, publishing flows, and freshness SLOs.
- Build experimentation infrastructure, including traffic splitting and reporting pipelines.
- Perform historical backfills and safe reprocessing after mapping changes.
- Implement data isolation and safe-aggregation controls for advertiser data protection.
- Develop automated data-quality validation frameworks.
- Collaborate with Customer engineers and prepare operational documentation.
- Contribute to architecture discussions and platform scalability improvements.
Requirements
- 5+ years of experience in Data Engineering.
- At least 2 years of experience working with production ML or large-scale analytics pipelines.
- Expert-level SQL skills, including window functions and incremental processing patterns.
- Strong Python skills for production-grade pipeline development.
- Hands-on experience with Spark or PySpark.
- Experience designing ETL/ELT pipelines with Airflow, Cloud Composer, Dagster, or similar tools.
- Experience with cloud data warehouses at scale, preferably BigQuery.
- Strong understanding of data modeling and point-in-time correctness.
- Experience with very large-scale event-driven or clickstream datasets and business-critical production pipelines.
- Upper-Intermediate English level or higher.
- Preferred experience with GCP services, including Dataflow, Pub/Sub, GCS, and Beam.
- Preferred experience building streaming or near-real-time ingestion systems.
- Preferred knowledge of feature stores, train/serve skew, label leakage prevention, delayed or incomplete ML labels, and experimentation infrastructure.
- Preferred experience with dbt, Customer-owned infrastructure, GDPR/CCPA-related privacy engineering, hybrid cloud/on-prem Linux environments, Terraform, Kubernetes, and warehouse cost and performance optimization.
- Strong analytical, problem-solving, communication, documentation, collaboration, and independent-working skills.
Benefits
- Opportunity to work on impactful global products and large-scale distributed systems.
- Collaboration with experienced engineers and participation in architecture decisions.
- Opportunity to grow expertise in modern data platforms, ML-oriented systems, and real-time AdTech infrastructure.
Tech Stack
Apache AirflowApache BeamApache SparkdbtGoogle BigQueryGoogle Cloud PlatformKubernetesLinuxPythonSQLTerraform
Categories
Data Engineering
About Sigma Software
Sigma Software is a London-headquartered, privately held IT consultancy that designs and builds custom software for startups and enterprises. The company provides end-to-end services—engineering, web and mobile development, .NET/Java platforms, AR/VR, and UI/UX—delivered via outsourcing, managed teams, and consulting. Founded in 2002, it supports long-term product development and modernization projects across multiple industries for global clients.