5 days ago
Amsterdam, NetherlandsStaff+
Responsibilities
- Design, implement, and operate web-scale crawling systems for acquiring internet content.
- Build ingestion workflows for crawlers, structured feeds, partner integrations, and other internal and external data sources.
- Develop crawl scheduling, prioritization, recrawl policies, freshness strategies, URL discovery, deduplication, content extraction, and crawl orchestration systems.
- Ensure reliable, high-throughput operation of crawling infrastructure processing billions of URLs.
- Define observability and quality metrics for crawl coverage, freshness, throughput, and content quality.
- Monitor resource usage, bandwidth consumption, and infrastructure cost.
- Collaborate with indexing and ML teams to meet retrieval and ranking requirements.
- Enable safe experimentation with crawling strategies and content acquisition policies.
Requirements
- 8+ years of experience building backend or distributed systems.
- Strong expertise in Go or C++.
- Experience with large-scale distributed systems handling 10k+ RPS, billions of URLs, or high-throughput pipelines.
- Understanding of HTTP, DNS, TLS, web crawling, scraping, and content extraction.
- Experience operating production systems and debugging failures in distributed environments.
- Strong understanding of scalability, fault tolerance, and resource management.
- Experience with web crawling, streaming data pipelines, event-driven systems, distributed schedulers, queues, asynchronous processing, or large-scale content platforms is valued.
- Experience with Kafka, Pulsar, NATS, RabbitMQ, Spark, Flink, Beam, or MapReduce is a strong plus.
- Coding interviews are part of the hiring process.
Benefits
- Competitive compensation.
- Career growth and learning opportunities.
- Flexibility, ownership, and a collaborative, innovative culture.
- Opportunity to work on impactful AI projects.
- International environment with talented teams.
Tech Stack
Categories
BackendData Engineering
About Nebius
Nebius builds a full-stack AI cloud offering GPU compute, storage, and tools for training and deploying ML models for startups, enterprises, and research labs. It sells consumption-based cloud infrastructure (IaaS/PaaS) and managed services tailored to generative AI workloads, including large-scale model training and inference. The company is headquartered in Amsterdam and operates as an independent provider.