[C3A] Senior Data Platform Engineer (Microsoft Fabric)
Software Mind3 days ago
Remote, EMEA or Warsaw, PolandSenior
Responsibilities
- Design and implement Bronze, canonical Silver, and Gold Delta table schemas with PAN-safe physical layouts and performance optimizations.
- Build Fabric Spark notebooks and feature pipelines for rolling, merchant-level, hierarchical, and cross-merchant risk features.
- Author versioned Silver adapters, canonical Bronze-to-Silver MERGE DAGs, and idempotent Gold materialization pipelines.
- Design Direct Lake semantic models and performance-test them against representative data volumes.
- Configure OneLake namespaces, workspace catalogs, and cross-workspace shortcut relationships.
- Implement idempotent Delta writes using delta-rs, segmented parallel writes, and atomic commits.
- Size Microsoft Fabric capacity and evaluate F-SKU, Reservation, and PAYG options through load testing.
- Partner with DevOps on Terraform capacity provisioning and Fabric REST workspace bootstrap.
- Advise on ClickHouse as a contingency for Direct Lake latency issues.
Requirements
- 5+ years of data engineering experience, including at least 2 years of hands-on Spark development; PySpark is preferred.
- Deep knowledge of Delta Lake internals, including transaction logs, checkpoints, retention, VACUUM, OPTIMIZE, deletion vectors, and statistics collection.
- Expertise in Parquet footers, statistics allowlists, row-group sizing, dictionary encoding, and column indexes.
- Hands-on Microsoft Fabric experience with workspaces, capacities, notebooks, lakehouses, semantic models, Direct Lake, and SQL analytics endpoints.
- SQL performance-tuning experience across OLAP Fabric and OLTP PostgreSQL workloads.
- Strong Python data engineering skills with pandas, PyArrow, and delta-rs.
- Ability to communicate architecture clearly in writing and work autonomously on well-scoped features.
- Preferred: PCI-scoped data platform experience and financial services or payments domain background.
- Preferred: Power BI semantic model authoring with DAX and familiarity with SCD2 dimensions, natural-key idempotency, extension attribute stores, and data-standard governance patterns.
- Preferred: OneLake shortcuts, cross-workspace lakehouse patterns, ClickHouse or similar OLAP engines, Terraform for Azure data resources, and fluid or schema-flexible ingestion patterns.
Benefits
- Flexible and remote cooperation options.
- International projects with leading global clients and travel related to international projects.
- Non-corporate atmosphere.
- Access to language classes, knowledge-sharing initiatives, private healthcare and life insurance, and a multisport card; availability and terms vary by cooperation model.
Tech Stack
Categories
Data Engineering
About Software Mind
Software Mind provides outsourced software engineering and autonomous, cross-functional development teams to startups through enterprises, delivering custom applications, cloud, AI, data and embedded solutions. The privately held company, headquartered in Krakow, operates across Europe, the US and LATAM, supporting full software lifecycles from ideation to maintenance. It offers nearshore/offshore engagement models and received private equity investment in 2020.