1 day ago
Pune, India or Bengaluru, IndiaSenior
Responsibilities
- Define and maintain testing methodologies across the GCP data engineering stack, including Dataflow, Dataproc, Cloud Run, BigQuery, and dbt.
- Develop data quality frameworks using dbt tests, Great Expectations, Soda Core, and Dataplex.
- Implement integration, regression, container-level, HTTP-triggered, and database tests for ETL/ELT pipelines and services.
- Use emulators or dedicated test instances to validate Spanner, Cloud SQL, AlloyDB, stored procedures, and database functions.
- Orchestrate end-to-end pipeline tests through Cloud Composer, Apache Airflow, or scripting.
- Embed automated QA and data quality validation into GitLab CI, Jenkins, and GitHub Actions pipelines.
- Provision reproducible test environments with Terraform and Deployment Manager.
- Partner with data engineers and stakeholders on testability, mentor junior QA team members, and improve data quality KPIs.
- Use Cloud Logging, Monitoring, and observability tools to track pipeline health, test results, and anomalies.
Requirements
- At least 5 years of QA or software testing experience focused on data pipelines or data warehouses.
- Proficiency in complex SQL and data validation queries.
- Strong hands-on experience with GCP data tools including dbt, BigQuery, Dataflow, Dataproc, Cloud Run, Spanner, AlloyDB, and Cloud SQL.
- Experience with Great Expectations, Soda Core, dbt-utils, and dbt-expectations.
- Familiarity with database emulators and test environment provisioning.
- CI/CD automation experience with Jenkins, GitLab CI, or GitHub Actions.
- Scripting experience with Python and pytest.
- Experience with API testing, Postman, and REST APIs.
- Strong communication, mentoring, critical-thinking, and attention-to-detail skills.
- Preferred: GCP Professional Data Engineer certification; experience with Dataplex, Cloud Logging, Elementary Data, data governance, lineage, compliance validation, data migration, or data warehouse modernization.
Tech Stack
Apache AirflowdbtGitHub ActionsGitLab CI/CDGoogle BigQueryGoogle Cloud PlatformJenkinsPostmanpytestPythonSQLTerraform
Categories
