Data Engineer
TechBiz Global GmbH11 months ago
Remote, WorldwideSenior
Responsibilities
- Design, develop, and maintain real-time and batch data ingestion pipelines using Kafka Connect and Debezium.
- Ingest data from MySQL and PostgreSQL into AWS S3, Google Cloud Storage, and BigQuery.
- Implement data modeling, schema evolution, and efficient partitioning practices in the Bronze Layer.
- Ensure the reliability, scalability, and monitoring of Kafka Connect clusters and connectors.
- Collaborate with cross-functional teams to understand source systems and downstream data requirements.
- Optimize ingestion processes for performance and cost efficiency.
- Contribute to automation and deployment scripts using Python and cloud-native tools.
- Stay current with data lake technologies such as Apache Hudi and Apache Iceberg.
Requirements
- 5+ years of hands-on experience as a Data Engineer or in a similar role.
- Strong experience with Apache Kafka and Kafka Connect, including sink and source connectors.
- Experience with Debezium for change data capture from relational databases.
- Proficiency with MySQL and PostgreSQL.
- Hands-on experience with AWS S3, GCP BigQuery, and Google Cloud Storage.
- Proficiency in Python for automation, data handling, and scripting.
- Understanding of data lake architectures and ingestion patterns.
- Understanding of ETL/ELT pipelines, data quality, and observability practices.
- Preferred experience with Docker and Kubernetes.
- Preferred familiarity with Airflow or Dagster for workflow orchestration.
- Preferred exposure to Terraform or CloudFormation for infrastructure as code.
- Preferred familiarity with data versioning and table technologies.
Tech Stack
Categories
Data Engineering
About TechBiz Global GmbH
TechBiz Global GmbH provides IT recruitment, outstaffing, and custom software development for startups and enterprises, plus consulting around product building and scaling. The privately held company is headquartered in Berlin and was founded in 2021. It staffs engineering roles for clients and delivers outsourced development teams, supporting engagements across multiple countries.