2 months ago
Jaipur, IndiaStaff+
Responsibilities
- Design and implement advanced web-scraping solutions using Python, Playwright, and AWS services.
- Automate interactions with JavaScript-heavy and authentication-secured websites, including MFA, CAPTCHAs, and session or token-based login flows.
- Architect serverless scraping pipelines using AWS Lambda, Step Functions, S3, CloudWatch, and Secrets Manager.
- Build scalable data-extraction systems with fault tolerance, retries, and intelligent logging.
- Integrate workflows across multiple portals, APIs, and data sources.
- Contribute to architecture, tooling, and engineering best practices.
- Perform code reviews and deployments.
Requirements
- Bachelor’s or master’s degree in computer science, software engineering, or a related field, or equivalent professional experience.
- 10+ years of Python development experience focused on automation and data extraction.
- Expertise with web-scraping tools including Playwright, Selenium, Scrapy, BeautifulSoup, and requests.
- Experience handling MFA, CAPTCHA solving, session management, cookie management, and multi-step authentication flows.
- Proficiency deploying and managing AWS workloads, particularly Lambda, S3, IAM, CloudWatch, and Secrets Manager.
- Experience with asynchronous programming, headless browsers, and JavaScript-rendered content.
- Understanding of HTTP, HTTPS, cookies, headers, and authentication or network-call reverse engineering.
- Experience with APIs, JSON/XML, and data transformation.
- Experience with CI or CD pipelines, Docker, and infrastructure as code using CloudFormation or Terraform.
- Familiarity with Apache Airflow or AWS Step Functions for data-pipeline orchestration.
- Experience scraping secure or enterprise-level financial, healthcare, or legal portals.
- Data engineering or ETL experience is a plus.
- Exposure to Python testing frameworks and unit or integration testing.
Benefits
- Competitive salary, annual performance bonus, and equity for full-time employees; compensation figures are not specified.
- Healthcare with 100% employer-paid health and dental insurance.
- Generous paid time off.
- Hybrid work arrangement: four days in the office and remote work on Fridays.
About iCapital
iCapital builds a technology platform that lets wealth managers and asset managers offer, transact, and service alternative investments and structured solutions for private clients. The company earns software, data, and administration fees across marketplace access, enterprise integrations, and fund servicing. Founded in 2013 and headquartered in New York, it reports $1.2 trillion of assets serviced globally and serves nearly 3,900 wealth management firms and 144,000 active financial professionals.