1 day ago
London, United KingdomSenior
Responsibilities
- Build and operate large-scale pipelines transferring video, LiDAR, and sensor data from global partners at up to petabyte-per-day scale.
- Design and evolve the globally distributed hub-and-spoke Data Transfer Hub.
- Build parallelized distributed transfer pipelines using Flyte, Kafka event-driven patterns, and Azure infrastructure.
- Write pipeline logic in Python and infrastructure-as-code in Terraform.
- Implement retry strategies, backfills, data integrity checks, lifecycle tracking, dashboards, and alerting.
- Develop cloud-to-cloud and cross-cloud networking solutions.
- Collaborate with product, commercial, and partner-facing teams to address customer and partner requirements.
- Participate in an out-of-hours on-call and support rotation after establishing operational foundations.
Requirements
- Proven experience building and operating large-scale data transfer, ingestion, or processing pipelines, ideally at multi-terabyte or petabyte scale.
- Strong software engineering skills and experience delivering and maintaining production systems.
- Experience with workflow orchestration for parallelized jobs; Flyte is preferred, with Airflow, Prefect, or similar tools also relevant.
- Experience building reliable and observable data systems with retries, backfills, integrity checks, lifecycle tracking, and operational alerting.
- Strong cloud platform skills, preferably with Azure; AWS or GCP experience is a plus.
- Proficiency in Python and Terraform.
- Experience with blob or object storage, large-scale file movement, networking, or cloud-to-cloud data transfer.
- A customer-focused product mindset and comfort working with evolving requirements.
- Experience with external data partners or technical integrations is desirable.
- Background in autonomous vehicles, robotics, geospatial data, video processing, or other data-intensive domains is desirable.
- Experience with Kafka or other event-driven systems, transfer-status dashboards, multi-cloud systems, or improving cloud-system cost, throughput, or reliability is desirable.
Benefits
- Full-time role based in London.
- Hybrid working policy combining time in Wayve offices and workshops with working from home.
- Inclusive interview experience with accommodations or adjustments available upon request.
Tech Stack
Categories
BackendData Engineering
