Member of Technical Staff (Web Access Engineer, Crawler)
Perplexity3 days ago
London, United KingdomSenior
Responsibilities
- Own fetching and rendering improvements end to end, partnering with crawler engineers and SREs through production rollout.
- Investigate failures across DNS, TLS, HTTP, caching, redirects, and client-side rendering, and classify root causes.
- Improve page loading, completion detection, and content capture for dynamic pages.
- Ensure the crawler respects access restrictions and rate limits, identifies itself clearly, and retries only transient technical errors.
- Build failure-reproduction tools, compare crawler output with expected page content, and create regression tests.
- Define and use metrics for fetching reliability, content completeness, and error classification.
Requirements
- 5+ years of software engineering experience in web infrastructure, browsers, CDNs, or crawling.
- Strong programming skills in at least one of Go, Python, Rust, or C++.
- Deep understanding of HTTP/1.1, HTTP/2, HTTP/3, TLS, DNS, cookies, sessions, caching, conditional requests, content encoding, redirects, and HTTP error handling.
- Knowledge of CDN caching, content delivery, retry backoff, and server-defined rate limits.
- Understanding of page lifecycle, network idle, service workers, client-side rendering, and shadow DOM.
- Experience debugging JavaScript and fluency with Chrome DevTools, CDP, HAR files, curl, and packet captures.
- Experience with CDN, caching, or web-performance systems; crawlers, headless browsers, or distributed content-acquisition systems at scale; or browser-engine and JavaScript-runtime contributions is preferred.
- Ability to work independently when the path forward is not fully defined.
About Perplexity
Perplexity builds an AI-powered answer engine for web and mobile and an enterprise product, Perplexity Computer, for AI-driven workflows across tools and apps. The company monetizes through individual subscriptions (Perplexity Pro) and enterprise plans. Founded in 2022 and headquartered in San Francisco, it operates as a privately held company.