8 days ago
Remote, BrazilSenior
Responsibilities
- Develop, test, and maintain data ingestion, transformation, and quality-check pipelines using PySpark/SparkSQL on Databricks.
- Build and modify Airflow DAGs using MWAA for pipeline orchestration.
- Write and optimize SQL queries for data transformation and validation.
- Implement and monitor data-quality checks using Great Expectations or an equivalent tool.
- Investigate pipeline failures and data-quality issues with guidance from senior engineers.
- Participate in code reviews, sprint planning, estimation, and retrospectives.
- Write and maintain documentation for datasets and pipelines.
- Follow team standards for CI/CD, testing, and data governance.
Requirements
- 3+ years of relevant experience with a Bachelor's or Associate’s Degree, or 5+ years of relevant experience.
- Proficiency in English at B2 level or above.
- Python experience for automation and data processing.
- Knowledge of batch data pipelines and ETL/ELT patterns.
- Familiarity with Apache Spark, including PySpark or SparkSQL.
- Experience with Databricks jobs, workflows, cluster management, and tuning.
- Intermediate-to-advanced SQL skills, including joins, window functions, and CTEs.
- Knowledge of Amazon S3 data lake design, including partitioning, layout, and lifecycle management.
- Basic AWS cloud concepts, including S3, IAM, and CloudWatch.
- Knowledge of dimensional/Kimball data modeling and medallion layers.
- Experience with Git/GitHub workflows and code review.
- Preferred: experience with Delta Lake, Apache Airflow/MWAA, CDC patterns such as DMS and incremental processing, data-quality validation and testing, Terraform, and BI integration with Superset.
Benefits
- Remote position; candidates must be based in Brazil, with possible scheduled presence at a Visa office when notified.
- Visa is an equal opportunity employer.
Categories
Data Engineering
About Visa
Visa builds and operates a global digital payments network that connects consumers, merchants, financial institutions, and governments. It earns fees for processing card transactions and offering risk, data, and value‑added services through VisaNet and products like tokenization and tap‑to‑pay. Founded in 1958 and headquartered in Foster City, California, Visa is a public company (NYSE: V) serving clients in more than 200 countries and territories.