1 month ago
Bengaluru, IndiaSenior
Responsibilities
- Design, build, and maintain scalable, reliable, and cost-efficient data pipeline architectures.
- Develop idempotent automated ETL/ELT pipelines using Python, PySpark, Delta Lake, and MERGE INTO patterns.
- Automate deployments, monitoring, incident remediation, and operational processes.
- Implement DataOps practices including data-contract enforcement, data-quality checks, unit testing, and integration testing.
- Establish data governance, lineage tracking, and fine-grained security access controls using Unity Catalog.
- Manage infrastructure as code for Databricks workspaces, compute clusters, serverless SQL warehouses, and storage integrations.
- Build and maintain CI/CD pipelines using Databricks Asset Bundles, Databricks CLI, GitHub Actions, Azure DevOps, or GitLab.
- Monitor compute consumption and costs, optimize cluster configurations, and implement cluster policies.
Requirements
- At least five years of experience in data engineering, DataOps, DevOps, or a comparable technical role.
- Advanced hands-on experience with Python, PySpark, SQL, Git, and Databricks.
- Strong experience with Delta Lake, Unity Catalog, and Databricks Serverless SQL.
- Experience with infrastructure as code, preferably Terraform.
- Experience with CI/CD tooling such as Databricks Asset Bundles, GitHub Actions, Azure DevOps, or GitLab.
- Experience implementing data-quality controls, data contracts, automated testing, monitoring, and incident-management practices.
- Strong knowledge of Azure cloud services, including security, networking, identity and access management, and storage integration.
- AWS or GCP experience is beneficial.
Categories
Data Engineering
About Bayer
Bayer is a global life-sciences company that develops and sells prescription pharmaceuticals, consumer health brands (including Aspirin), and agricultural seeds, traits, and crop-protection products, plus digital farming tools, for patients, consumers, and farmers. It is a public company founded in 1863 and headquartered in Leverkusen, Germany. Bayer acquired Monsanto in 2018, making Crop Science one of its three core businesses alongside Pharmaceuticals and Consumer Health.