1 month ago
Remote, EMEA +2 moreMid Level
Responsibilities
- Migrate approximately 1,000 datasets and pipelines from Apache Airflow 2 to Apache Airflow 3.
- Migrate Confluent connections between cloud regions.
- Migrate pipelines from Hive to Iceberg using Snowpark Connect.
- Adapt existing data pipelines to Snowflake while maintaining data quality and reliability.
- Support workflow scheduling and monitoring in Apache Airflow.
- Execute time-sensitive data migrations and collaborate with the team on implementation delivery.
- Develop, test, debug, and maintain data pipelines using Python and Apache Airflow.
Requirements
- At least 2 years of industry experience working with large-scale datasets, including Airflow and Spark.
- Strong software engineering fundamentals and hands-on Python development experience.
- Advanced SQL skills, including complex joins, CTEs, window functions, and performance tuning.
- Experience with Snowflake data modeling.
- Experience developing and testing pipelines in Apache Airflow and Python.
- Experience with Apache Kafka, Apache Spark, and hands-on PySpark.
- Experience with Iceberg and Parquet data formats.
- AWS experience, particularly S3 data lakes, partitioning, and lifecycle policies.
- Experience with testing strategies and hands-on debugging and code testing.
- Strong communication skills and the ability to work independently.
- Enterprise experience is strongly preferred.
- Snowflake migration, data warehouse modernization, performance and cost optimization, and Airflow migration experience are nice-to-have qualifications.
- Experience using AI tools for development is a nice-to-have qualification.
Benefits
- Remote contract engagement for 40 hours per week.
- Open to candidates in eligible EMEA and APAC locations.
- Coverage until 12:00 PM Pacific Time is required.
- Expected engagement through March 31, 2027, with the start date to be confirmed.
- This is not currently a contract-to-hire opportunity.
Tech Stack
Categories
Data Engineering
About Lifted
Lifted builds a global platform for enterprises to source, contract, manage, and pay contingent workers with embedded compliance. The software integrates into existing contingent workforce ecosystems and supports multiple worker types across countries. Lifted operates as an Upwork company, serving organizations that need to centralize contingent programs and streamline cross‑border payments and documentation.