Senior Data Engineer - Data Engineering
GEP Worldwide (NB Ventures INC)15 days ago
São Paulo, BrazilSenior
Responsibilities
- Lead the design, development, and optimization of scalable Databricks data solutions using Python, SQL, and PySpark.
- Architect Delta Lake lakehouse and medallion solutions, including data models, layer boundaries, and performance strategies.
- Define and govern Unity Catalog assets, permissions, schemas, catalogs, and lineage.
- Build batch and incremental pipelines with Lakeflow Jobs, Lakeflow Declarative Pipelines, MERGE/CDC patterns, and data-quality expectations.
- Own CI/CD packaging and deployment across environments using Databricks Asset Bundles, Git, and automated testing.
- Establish data-quality, observability, and data-contract practices for business-critical datasets.
- Optimize Databricks workloads for performance and cost through compute selection, Spark tuning, and Delta table maintenance.
- Provide technical leadership in architecture, code reviews, troubleshooting, interviews, and mentoring engineers.
- Collaborate with analysts, product managers, and business stakeholders on data solutions and technical trade-offs.
Requirements
- Expert hands-on experience with Databricks, including Lakeflow Jobs, Lakeflow Declarative Pipelines, Spark DataFrames, and Delta Lake.
- Deep knowledge of Spark, distributed data processing, partitioning, joins, shuffles, and performance tuning.
- Expert Python and SQL skills for data engineering, including PySpark, Pandas, and production-grade testable code.
- Strong experience with Unity Catalog governance, CI/CD, Databricks Asset Bundles, Git workflows, and automated testing such as pytest.
- Knowledge of data modeling, lakehouse and medallion architecture, incremental processing, MERGE, CDC, and Delta Lake optimization.
- Experience designing data-quality frameworks, observability practices, and data contracts.
- Strongly preferred experience with Azure Databricks; AWS, Google Cloud, or another cloud platform is valuable.
- Familiarity with AI-assisted development tools, Databricks AI Functions, and web scraping tools such as BeautifulSoup, Scrapy, or Selenium is a plus.
- Knowledge of procurement, public procurement, or foreign trade data is a plus.
- Ability to lead projects in agile environments and communicate effectively with engineers and business stakeholders.
Benefits
- GEP describes opportunities for career growth, recognition, and advancement.
- GEP emphasizes an inclusive and diverse workplace and is an equal opportunity employer.
Tech Stack
Categories
Data Engineering
About GEP Worldwide (NB Ventures INC)
GEP Worldwide builds cloud software for procurement and supply chain operations and provides strategy consulting and managed services to large enterprises. Its platform covers sourcing, spend analysis, contract and supplier management, and supply chain planning, delivered as SaaS alongside implementation and outsourcing support. Founded in 1999 and headquartered in Clark, New Jersey, the privately held company serves multinational corporations across industries.