Senior Data Engineer

3 days ago

Remote, EMEASenior

Base Salary

$100k - $140k/yr

Responsibilities

Design and own a multi-source data platform for concurrent ingestion, event-driven processing, and orchestrated batch workflows.
Build resilient ingestion from provider APIs, webhooks, files, and web scraping with rate limiting and failure handling.
Orchestrate scheduled workflows and historical backfills with retries and recovery.
Develop event-driven processing with messaging or task queues, dead-letter queues, and controlled replay.
Store source payloads in Google Cloud Storage and transform them with dbt into raw, standardized, and curated layers.
Implement idempotent upserts, data contracts, automated validation, pipeline testing, and monitoring for freshness and failures.
Design and optimize PostgreSQL schemas and loading patterns.
Integrate data enrichment and LLM-based structured extraction into the platform.

At least 5 years of experience building and operating production data platforms that ingest data from multiple sources.
Production experience with workflow orchestration, dependencies, retries, and failure recovery; Airflow is preferred, with Dagster, Prefect, Temporal, or comparable tools also relevant.
Hands-on production experience with dbt and Google Cloud Platform.
Strong Python skills, including asynchronous HTTP and concurrent processing with asyncio, aiohttp, or HTTPX.
Strong SQL and PostgreSQL skills, including data modeling, query optimization, and incremental loading.
Experience with event-driven systems, queues, data contracts, raw and curated storage, retention, testing, and monitoring.
Experience building a platform or substantial pipelines from scratch is valued.
Experience with web scraping, entity resolution, enrichment, LLM-based extraction, or startup and small-team environments is useful.

Fully remote for candidates based in Europe, including the United Kingdom, Iceland, Liechtenstein, Norway, Switzerland, and Ukraine.