4 days ago
Bengaluru, IndiaSenior
Responsibilities
- Build reliable and scalable batch and streaming pipelines with PySpark and Databricks Workflows/Jobs.
- Implement Delta Lake schema enforcement and evolution, ACID transactions, time travel, OPTIMIZE/ZORDER, and VACUUM.
- Develop Delta Live Tables or Structured Streaming pipelines using Auto Loader.
- Write modular Python code with packaging, logging, configuration, and error handling.
- Write scalable SQL queries for extraction and reporting.
- Optimize Spark jobs using partitioning, join optimization, bucketing, caching, AQE, broadcast hints, and file sizing.
- Manage SQL Warehouses, Photon, Databricks Runtime versions, cluster policies, and pools.
- Configure workflow dependencies, parameters, retries, alerts, and schedules.
- Administer Unity Catalog permissions, lineage, auditing, external locations, and shares.
- Build idempotent, replayable, and incremental pipelines with checkpoints and watermarks.
- Implement data quality checks and unit and integration tests for pipelines.
- Automate deployment and testing of data workflows with CI/CD tools.
- Collaborate with stakeholders on data SLAs, schemas, and interface contracts.
- Document pipelines, datasets, data dictionaries, lineage, and usage guidelines.
Requirements
- 5–7 years of data engineering experience.
- Bachelor of Technology or equivalent BE/BTech education.
- Hands-on experience with Databricks Data Engineering, PySpark, Python, SQL, and data modeling.
- Hands-on knowledge of Azure Data Factory and Azure Data Lake Storage.
- Experience with production-grade ETL/ELT, orchestration, performance optimization, and data governance.
- Knowledge of Delta Lake, Databricks Workflows/Jobs, Delta Live Tables, Structured Streaming, Auto Loader, SQL Warehouses, Photon, and Unity Catalog.
- Experience with data quality practices, testing, CI/CD, and pipeline documentation.
- Databricks Certified Data Engineer Associate certification is mandatory.
- Analytical, problem-solving, communication, and stakeholder collaboration skills are preferred.
Benefits
- Inclusive benefits, wellbeing support, flexibility programs, mentorship, and professional growth opportunities.
- Equal employment opportunity and a workplace focused on inclusion and respectful collaboration.
Categories
Data Engineering
About PwC
PwC is a global network of professional services firms providing assurance, tax, and advisory work for businesses and public-sector clients. It sells project-based and managed consulting, audit, and deals services, and implements and integrates business applications. Formed in 1998 by the merger of Price Waterhouse and Coopers & Lybrand, it is headquartered in London and is one of the Big Four accounting firms.
