3 days ago
Zürich, SwitzerlandStaff+
Responsibilities
- Architect and scale data acquisition infrastructure for foundation-model training, including source discovery, URL indexing, crawl frontiers, scheduling, fetching, and ingestion.
- Design high-throughput frontier storage and queueing systems with deduplication, efficient lookups, and quality-, diversity-, and freshness-aware scheduling.
- Develop shared crawl execution and extensible pipelines for open-web, licensed, partner, text, PDF, code, and multimodal content.
- Partner with parsing, curation, research, product, storage, scaling, legal, privacy, security, and governance teams to improve data quality and enforce approved usage controls.
- Own production readiness, capacity planning, observability, fault tolerance, recovery, cost efficiency, and service objectives for throughput, freshness, and reliability.
- Lead architecture reviews, critical implementation work, cross-team dependency coordination, technical tradeoffs, mentoring, and team building.
- Evaluate crawls and datasets through reproducible pilots and deliver data releases connected to controlled training experiments.
Requirements
- BS, MS, or PhD in Computer Science, Computer Engineering, or a related technical field, or equivalent experience.
- 15+ years of relevant software engineering experience designing and operating large-scale distributed systems, setting technical direction, and delivering complex infrastructure across teams.
- Deep experience with web crawling or related discovery, indexing, or ingestion infrastructure, including distributed scheduling, stateful services, storage, and high-throughput processing.
- Strong hands-on engineering and system design skills spanning architecture, implementation, performance analysis, and production debugging.
- Experience building batch or streaming pipelines and handling consistency, retries, duplicate processing, backpressure, recovery, and operational efficiency.
- Experience delivering data platforms for research, machine learning, search, or other demanding technical users.
- Strong technical communication and leadership through influence across engineering, product, research, and governance teams.
- Preferred experience includes web-wide crawl platforms, foundation-model pretraining data, publisher consent and licensing controls, provenance infrastructure, browser-based or agentic collection, programmable extraction, sandboxed environments, licensed or multimodal datasets, partner integrations, or low-latency streaming feeds.
Categories
BackendData Engineering
About Nvidia
Nvidia designs and sells GPUs and accelerated computing platforms for data centers, AI/ML, graphics, gaming, and automotive, monetizing through hardware, software platforms (CUDA, AI frameworks), and systems like DGX and networking. Customers include cloud providers, enterprises, researchers, and OEMs. Founded in 1993 and headquartered in Santa Clara, it is a public company traded on NASDAQ under NVDA.