PwC

IN_Senior Associate_ Databricks Senior Data Engineer_GCC_Advisory_Bangalore

PwC
Apply
4 days ago
Bengaluru, IndiaSenior

Responsibilities

  • Build reliable and scalable batch and streaming pipelines with PySpark and Databricks Workflows/Jobs.
  • Implement Delta Lake schema enforcement and evolution, ACID transactions, time travel, OPTIMIZE/ZORDER, and VACUUM.
  • Develop Delta Live Tables or Structured Streaming pipelines using Auto Loader.
  • Write modular Python code with packaging, logging, configuration, and error handling.
  • Write scalable SQL queries for extraction and reporting.
  • Optimize Spark jobs using partitioning, join optimization, bucketing, caching, AQE, broadcast hints, and file sizing.
  • Manage SQL Warehouses, Photon, Databricks Runtime versions, cluster policies, and pools.
  • Configure workflow dependencies, parameters, retries, alerts, and schedules.
  • Administer Unity Catalog permissions, lineage, auditing, external locations, and shares.
  • Build idempotent, replayable, and incremental pipelines with checkpoints and watermarks.
  • Implement data quality checks and unit and integration tests for pipelines.
  • Automate deployment and testing of data workflows with CI/CD tools.
  • Collaborate with stakeholders on data SLAs, schemas, and interface contracts.
  • Document pipelines, datasets, data dictionaries, lineage, and usage guidelines.

Requirements

  • 5–7 years of data engineering experience.
  • Bachelor of Technology or equivalent BE/BTech education.
  • Hands-on experience with Databricks Data Engineering, PySpark, Python, SQL, and data modeling.
  • Hands-on knowledge of Azure Data Factory and Azure Data Lake Storage.
  • Experience with production-grade ETL/ELT, orchestration, performance optimization, and data governance.
  • Knowledge of Delta Lake, Databricks Workflows/Jobs, Delta Live Tables, Structured Streaming, Auto Loader, SQL Warehouses, Photon, and Unity Catalog.
  • Experience with data quality practices, testing, CI/CD, and pipeline documentation.
  • Databricks Certified Data Engineer Associate certification is mandatory.
  • Analytical, problem-solving, communication, and stakeholder collaboration skills are preferred.

Benefits

  • Inclusive benefits, wellbeing support, flexibility programs, mentorship, and professional growth opportunities.
  • Equal employment opportunity and a workplace focused on inclusion and respectful collaboration.

Tech Stack

Apache AirflowApache HadoopApache SparkAWSAzureDatabricksGitJenkinsPythonSQL

Categories

Data Engineering
PwC

About PwC

10,000+ employees

PwC is a global network of professional services firms providing assurance, tax, and advisory work for businesses and public-sector clients. It sells project-based and managed consulting, audit, and deals services, and implements and integrates business applications. Formed in 1998 by the merger of Price Waterhouse and Coopers & Lybrand, it is headquartered in London and is one of the Big Four accounting firms.

Contact me