[C3F] Data Platform Engineer
Software Mind5 hours ago
Remote, Poland or Warsaw, PolandSenior
Responsibilities
- Build and operate change-data-capture pipelines from PostgreSQL into Azure using Kafka Connect and Debezium.
- Configure, deploy, scale, and recover connectors, including task management, offsets, schema history, and snapshot strategies.
- Run CDC pipelines as stateful Kubernetes workloads on AKS and manage configuration, secrets, networking, and resource tuning.
- Monitor and troubleshoot production connector failures, task rebalances, restarts, throughput, backpressure, message-size limits, retries, and recovery.
- Automate data-platform onboarding, orchestration, monitoring, alerting, recovery workflows, and testing with Python.
- Integrate CDC streams with Event Hubs, ADLS, Azure PostgreSQL, ADF, and Databricks.
- Manage reproducible, reviewable platform infrastructure as code and apply masking, hashing, access control, and retention controls to sensitive data.
Requirements
- Solid commercial experience as a data or platform engineer with hands-on streaming or CDC pipeline experience.
- Practical Kafka knowledge covering topics, partitions, offsets, consumer groups, delivery semantics, and at least one self-operated Kafka Connect deployment.
- Strong SQL and PostgreSQL skills, including WAL, logical replication, replication slots, and replication lag.
- Working knowledge of CDC concepts including snapshots, inserts, updates, deletes, event ordering, at-least-once delivery, and schema evolution.
- Confident Python skills for orchestration, monitoring, recovery tooling, and automated tests.
- Hands-on experience with Azure data services such as Event Hubs, ADLS, or Azure PostgreSQL.
- Experience using Kubernetes to deploy workloads, handle configuration and secrets, read logs, and debug failing pods.
- Ability to diagnose running pipelines using metrics and logs, distinguishing throughput issues, backpressure, retries, and connector failures.
- Production experience with Debezium, including large-table snapshots, schema-history recovery, offset loss, and connector restoration.
- Experience operating stateful workloads on AKS, including StatefulSets, stable worker identity, and resource tuning under load.
- Infrastructure-as-code and CI/CD experience for data-platform components using Terraform, Bicep, or similar tools.
- Hands-on production-scale experience with Databricks and ADF, plus data protection controls for sensitive data.
Benefits
- Flexible and remote cooperation options.
- International projects with leading global clients and travel related to international projects.
- Non-corporate atmosphere.
- Access to language classes and knowledge-sharing initiatives.
- Access to private healthcare, life insurance, and a multisport card.
- Benefit availability and terms may vary depending on the chosen form of cooperation.
Tech Stack
Categories
Data EngineeringDevOps
About Software Mind
Software Mind provides outsourced software engineering and autonomous, cross-functional development teams to startups through enterprises, delivering custom applications, cloud, AI, data and embedded solutions. The privately held company, headquartered in Krakow, operates across Europe, the US and LATAM, supporting full software lifecycles from ideation to maintenance. It offers nearshore/offshore engagement models and received private equity investment in 2020.