about 2 hours ago
Tel Aviv-Yafo, IsraelMid Level / Senior
H1B Sponsor
Responsibilities
- Own the lifecycle of data pipelines running in 15+ regions.
- Write big, distributed processing jobs for massive datasets.
- Build and maintain pipelines for the product's access layer.
- Implement frameworks to check data health and run conditional checks.
- Ensure observability by implementing distributed tracing.
Requirements
- 4+ years of experience building production data pipelines at scale.
- Proficiency in Python 3.10+ including Pydantic and async.
- Experience with Apache Airflow for DAG authoring.
- Knowledge of PySpark for job authoring and optimization.
- Expertise in Snowflake, including Snowpark and schema design.
- Familiarity with AWS services like S3 and EMR.
- Experience with data quality tooling such as Great Expectations.
- Strong CI/CD experience with artifact packaging and staged deployments.
- Solid testing discipline using pytest and integration tests.