1 month ago
Bengaluru, IndiaMid Level
Responsibilities
- Design, build, maintain, and optimize scalable batch and streaming data pipelines using Apache Spark and cloud-native data technologies.
- Develop ETL/ELT workflows to ingest, transform, and curate data from diverse source systems into analytics-ready datasets.
- Implement data modeling and transformation logic for reporting, dashboards, analytical workloads, and machine learning use cases.
- Build and manage data processing workloads in lakehouse platforms, including Microsoft Fabric and OneLake.
- Implement data validation, monitoring, reconciliation, schema management, and controls for data quality and reliability.
- Develop reusable frameworks, libraries, and standardized data engineering patterns.
- Implement CI/CD pipelines for automated testing, deployment, and rollback of data workloads.
- Monitor pipelines, troubleshoot failures, and resolve performance and data quality issues.
- Collaborate with analytics engineers, BI developers, data scientists, product teams, platform partners, security teams, and governance teams.
- Participate in Agile delivery activities such as sprint planning, design reviews, and continuous improvement.
Requirements
- Strong experience building and operating production data pipelines as a Data Engineer.
- Hands-on experience with Apache Spark for large-scale data processing.
- Proficiency in Python, SQL, and data transformation practices.
- Experience with cloud-based data platforms and storage, including data lake or lakehouse architectures.
- Experience designing and optimizing data models for analytical workloads.
- Understanding of distributed data processing, performance tuning, and fault tolerance.
- Experience with CI/CD, version control, and infrastructure-as-code concepts.
- Strong problem-solving, troubleshooting, communication, and cross-functional collaboration skills.
- Four or more years of experience in data engineering or an equivalent role is preferred.
- Preferred experience includes Microsoft Fabric, OneLake or similar analytics platforms, real-time or event-driven processing, data governance, metadata management, data quality frameworks, orchestration tools, and analytical, reporting, or machine learning use cases.
Tech Stack
Categories
Data Engineering
About Allstate
Allstate is a major U.S. personal-lines insurer that sells auto, homeowners, renters, and life policies to consumers through agents and direct online channels. It earns premiums and service fees across insurance and protection products, including telematics-based programs and device/identity protection. Founded in 1931 and headquartered in Northbrook, Illinois, Allstate is publicly traded on the NYSE under the ticker ALL.