GEP Worldwide (NB Ventures INC)

Senior Data Engineer - Data Engineering

GEP Worldwide (NB Ventures INC)
Apply
20 hours ago
São Paulo, BrazilSenior

Responsibilities

  • Lead the design, development, and optimization of scalable web scrapers for large-scale data extraction and analytics.
  • Architect scalable, secure, and efficient scraping solutions and data workflows aligned with business goals.
  • Collaborate with analysts, product managers, and business leaders to define scraping strategies and deliver actionable insights.
  • Implement data validation and monitoring frameworks to improve the accuracy and consistency of scraped data.
  • Drive best practices in web scraping and data pipeline management, including CI/CD and workflow automation.
  • Mentor and guide junior web-scraping engineers.
  • Evaluate and integrate scraping and workflow technologies such as Apache Airflow.
  • Provide technical leadership in troubleshooting critical scraping and workflow issues and optimizing system performance.

Requirements

  • Bachelor’s or master’s degree in Computer Engineering, Computer Science, Information Technology, or a related field, or equivalent experience.
  • Strong knowledge of data engineering and experience designing and leading data infrastructure projects.
  • Expert-level proficiency in Python for data processing and automation.
  • Expertise with web-scraping frameworks such as Scrapy and Selenium and with data modeling for complex extractions.
  • Knowledge of AWS, Google Cloud, or Azure and advanced data warehousing solutions.
  • Mastery of anti-scraping techniques, distributed scraping using proxies or headless browsers, and ethical compliance.
  • Strong experience with Apache Airflow for orchestrating and automating data pipelines, including scheduling and monitoring web-scraping workflows.
  • Eagerness to learn Big Data tools such as Spark and Databricks.
  • Ability to lead projects in agile environments while balancing technical and business priorities.
  • Excellent communication and mentorship skills; fluent spoken and written English is required, and Spanish is a plus.

Tech Stack

Apache AirflowApache SparkAWSAzureDatabricksGoogle CloudPythonSelenium

Categories

Data Engineering
GEP Worldwide (NB Ventures INC)

About GEP Worldwide (NB Ventures INC)

5,001-10,000 employees

GEP Worldwide builds cloud software for procurement and supply chain operations and provides strategy consulting and managed services to large enterprises. Its platform covers sourcing, spend analysis, contract and supplier management, and supply chain planning, delivered as SaaS alongside implementation and outsourcing support. Founded in 1999 and headquartered in Clark, New Jersey, the privately held company serves multinational corporations across industries.

Contact me