2 days ago
Zürich, SwitzerlandSenior
Base Salary
$180k - $350k/yr
Responsibilities
- Synthesize web crawls, vendor datasets, APIs, and other sources into a typed knowledge graph with provenance, timestamps, and confidence.
- Build versioned, append-only ingestion pipelines supporting schema evolution, late and corrected records, replayable cursors, and source lineage.
- Develop entity resolution and truth discovery systems across billions of records to create canonical entities and reconcile conflicting claims.
- Build incremental computation over change data feeds with idempotent, resumable jobs and eventually consistent graph updates.
- Measure extraction and resolution quality through labeled evaluations, source coverage and correctness metrics, and failure-point lineage.
Requirements
- Experience building and operating large-scale distributed data pipelines in production over petabytes of structured and unstructured data.
- Experience using Spark with a lakehouse format such as Delta Lake, Iceberg, or Hudi.
- Deep understanding of incremental processing, change data capture, idempotent writes, backfills, and reprocessing.
- Strong Python skills and proficiency in at least one systems language.
- Ability to read and tune Spark query plans.
- Strong focus on correctness, reliability, data quality, and scale.
- Bonus experience with entity resolution, record linkage, knowledge graphs, ontologies, information extraction, truth discovery, Lance, or other columnar/vector-native formats.
Tech Stack
Categories
Data Engineering
About Exa
Exa builds a web-scale search engine and API for AI agents and developer applications, combining its own crawler, embedding models, and high-performance vector search. It sells usage-based APIs and enterprise integrations that power retrieval-augmented generation, browsing, and automation on live web content. The company runs large crawling infrastructure and dedicated GPU clusters to continuously index and embed pages, and ships low-latency search components written in Rust.