Software Lead- Data Engineering
GEP Worldwide (NB Ventures INC)7 months ago
São Paulo, BrazilStaff+
Responsibilities
- Lead the design and implementation of cloud-native data pipelines using Azure Functions, Data Factory, Blob Storage, and Container Instances.
- Architect resilient, large-scale web scraping and orchestration systems using Apache Airflow.
- Develop interactive Streamlit dashboards and internal data applications for data quality, pipeline health, and market-trend visibility.
- Establish engineering standards for Python development, CI/CD, data modeling, observability, and production reliability.
- Evaluate Spark, Databricks, and LLM-based parsing technologies to improve data accuracy and address anti-scraping measures.
- Define technical strategy and roadmaps while balancing technical debt with business delivery needs.
- Lead code reviews, technical career development, mentorship, and the team’s overall technical growth.
- Oversee data validation, data privacy, GDPR/LGPD compliance, and global ethical standards.
- Present technical solutions as business value to stakeholders and clients.
Requirements
- Bachelor’s or master’s degree in Computer Science, Engineering, or a related field.
- Knowledge of data engineering and experience designing and leading data infrastructure projects, with previous technical leadership or architecture experience.
- Experience building and deploying production-grade data applications.
- Advanced Python knowledge, including asynchronous programming with asyncio, automation, and data processing.
- Deep Azure knowledge, including Azure Container Registry, Azure Key Vault, and Managed Identities.
- Proven experience building and deploying Streamlit applications.
- Expertise with Scrapy, Selenium, Playwright, and complex proxy rotation.
- Expert-level Apache Airflow experience for complex DAG dependencies and scheduling.
- Experience with Docker, Kubernetes, and CI/CD pipelines using GitHub Actions or Azure DevOps.
- Fluency in English and Portuguese is required; Spanish is a plus.
- Strong strategic thinking, stakeholder management, global communication, and technical presentation skills.
Tech Stack
Categories
Data Engineering