
Member of Technical Staff (Index Engineer, Search Crawler)
Perplexity1 day ago
London, United KingdomSenior
Responsibilities
- Design, implement, and operate large-scale indexing systems and data pipelines for search infrastructure.
- Develop indexing strategies that balance performance, freshness, and resource efficiency.
- Build storage formats, compaction, and update mechanisms for current and accessible data.
- Improve pipeline reliability and predictability under high-throughput conditions.
- Build well-tested components with clear responsibilities and flexible interface contracts.
- Define observability for offline and nearline pipelines using structured logs, metrics, and data quality signals.
- Monitor throughput, resource usage, and cost, and drive optimizations.
- Partner with retrieval, ranking, and ML teams to meet downstream indexing requirements.
- Enable safe experimentation through controlled rollouts and clearly defined quality signals.
Requirements
- At least 5 years of experience building production backend or data infrastructure systems.
- Strong programming experience in Go, C++, or Rust.
- Experience with high-throughput data processing systems and very large datasets.
- Experience building or operating databases, storage systems, data planes, or indexing pipelines.
- Strong understanding of distributed systems, fault tolerance, consistency, and scalability.
- Experience running production systems and handling operational incidents.
- Ability to reason about end-to-end data flows with a systems-thinking mindset.
About Perplexity
Perplexity builds an AI-powered answer engine for web and mobile and an enterprise product, Perplexity Computer, for AI-driven workflows across tools and apps. The company monetizes through individual subscriptions (Perplexity Pro) and enterprise plans. Founded in 2022 and headquartered in San Francisco, it operates as a privately held company.