
Senior Data Engineer - Data Engineering
GEP Worldwide (NB Ventures INC)20 hours ago
São Paulo, BrazilSenior
Responsibilities
- Lead the design, development, and optimization of scalable web scrapers for large-scale data extraction and analytics.
- Architect scalable, secure, and efficient scraping solutions and data workflows aligned with business goals.
- Collaborate with analysts, product managers, and business leaders to define scraping strategies and deliver actionable insights.
- Implement data validation and monitoring frameworks to improve the accuracy and consistency of scraped data.
- Drive best practices in web scraping and data pipeline management, including CI/CD and workflow automation.
- Mentor and guide junior web-scraping engineers.
- Evaluate and integrate scraping and workflow technologies such as Apache Airflow.
- Provide technical leadership in troubleshooting critical scraping and workflow issues and optimizing system performance.
Requirements
- Bachelor’s or master’s degree in Computer Engineering, Computer Science, Information Technology, or a related field, or equivalent experience.
- Strong knowledge of data engineering and experience designing and leading data infrastructure projects.
- Expert-level proficiency in Python for data processing and automation.
- Expertise with web-scraping frameworks such as Scrapy and Selenium and with data modeling for complex extractions.
- Knowledge of AWS, Google Cloud, or Azure and advanced data warehousing solutions.
- Mastery of anti-scraping techniques, distributed scraping using proxies or headless browsers, and ethical compliance.
- Strong experience with Apache Airflow for orchestrating and automating data pipelines, including scheduling and monitoring web-scraping workflows.
- Eagerness to learn Big Data tools such as Spark and Databricks.
- Ability to lead projects in agile environments while balancing technical and business priorities.
- Excellent communication and mentorship skills; fluent spoken and written English is required, and Spanish is a plus.
Tech Stack
Categories
Data Engineering
About GEP Worldwide (NB Ventures INC)
GEP Worldwide builds cloud software for procurement and supply chain operations and provides strategy consulting and managed services to large enterprises. Its platform covers sourcing, spend analysis, contract and supplier management, and supply chain planning, delivered as SaaS alongside implementation and outsourcing support. Founded in 1999 and headquartered in Clark, New Jersey, the privately held company serves multinational corporations across industries.