2 months ago
Responsibilities
- Build and design large scale, distributed crawling bots and infrastructure.
- Develop and maintain data pipelines to extract data from various sources.
- Help unify heterogeneous documents into a coherent data schema.
- Preprocess and normalize raw data for classification and search indexing.
- Build APIs to expose structured, classified data via ElasticSearch/OpenSearch.
- Collaborate with ML/NLP teams to integrate classification models.
- Automate workflows using Apache Airflow and deploy solutions in Kubernetes.
- Optimize and scale data pipelines using Spark for large datasets.
Requirements
- 5+ years of experience in Python with building crawling/scraping solutions.
- Experience working with APIs and PDF processing.
- Proficiency in data processing and search technologies.
- Hands-on experience with Airflow and Spark or similar systems.
- Strong problem-solving skills in handling anti-scraping mechanisms.
- Hands-on experience with AWS or GCP.
Benefits
- Enjoy the flexibility of remote work.
- Competitive base salaries that reflect your value.
- Generous Paid Time Off for well-rested performance.
- Comprehensive health benefits with 100% premium coverage for employees.
- Free, confidential counseling through BetterHelp.
- 401(k) plan with 100% employer match on contributions.
- Up to $60/month for wellness-related expenses.
- Home office setup bonus after one year of employment.
- Support for continuing education and professional development.
Tech Stack
Apache AirflowApache SparkAWSElasticsearchGoogle Cloud PlatformKubernetesPythonPyTorchspaCySQLTensorFlow
Categories
AI & MLData Engineering