2 months ago
Base Salary
$150k - $180k/yr
Responsibilities
- Design and implement advanced web-scraping solutions using Python, Playwright, and AWS services.
- Automate interactions with JavaScript-heavy and authentication-secured websites, including MFA, CAPTCHAs, and session or token-based login flows.
- Architect scalable scraping pipelines using AWS Lambda, Step Functions, S3, CloudWatch, and Secrets Manager.
- Build high-volume data-extraction systems with fault tolerance, retries, and intelligent logging.
- Integrate workflows across multiple portals, APIs, and data sources.
- Contribute to architectural decisions, tooling, and engineering best practices.
Requirements
- Bachelor’s or master’s degree in computer science, software engineering, or a related field, or equivalent professional experience.
- 10+ years of Python development experience focused on automation and data extraction.
- Expertise with web-scraping tools including Playwright, Selenium, Scrapy, BeautifulSoup, and requests.
- Experience handling multi-step authentication flows, MFA, CAPTCHA solving, and session or cookie management.
- Proficiency deploying and managing workloads in AWS, especially Lambda, S3, IAM, CloudWatch, and Secrets Manager.
- Experience with asynchronous programming, headless browsers, JavaScript-rendered content, web protocols, and reverse-engineering network calls and authentication mechanisms.
- Experience with APIs, JSON/XML, data transformation, CI/CD pipelines, Docker, and infrastructure as code such as CloudFormation and Terraform.
- Familiarity with Apache Airflow or AWS Step Functions for data-pipeline orchestration.
- Experience scraping secure or enterprise-level financial, healthcare, or legal portals.
- Data engineering or ETL experience is a plus.
- Exposure to Python testing frameworks and unit or integration testing.
Benefits
- Base salary range of $150,000 to $180,000, plus equity for full-time employees and an annual performance bonus.
- Employer-matched retirement plan and subsidized healthcare with employer-paid dental and vision coverage.
- Telemedicine, virtual mental health counseling, parental leave, and unlimited paid time off.
- Hybrid schedule with office work Monday through Thursday and remote flexibility on Friday.
About iCapital
iCapital builds a technology platform that lets wealth managers and asset managers offer, transact, and service alternative investments and structured solutions for private clients. The company earns software, data, and administration fees across marketplace access, enterprise integrations, and fund servicing. Founded in 2013 and headquartered in New York, it reports $1.2 trillion of assets serviced globally and serves nearly 3,900 wealth management firms and 144,000 active financial professionals.