Lead Senior Associate | Data Engineering | Bengaluru | Engineering as a Service/ Operate
Deloitte Advisory1 month ago
Bengaluru, IndiaEntry Level
Responsibilities
- Design, maintain, optimize, and support Spark and Flink pipelines running in Databricks and Airflow.
- Ensure production data workflows are reliable, available, performant, and cost-efficient.
- Debug and resolve issues in notebooks, jobs, and orchestration workflows.
- Tune Spark jobs, clusters, and pipelines by analyzing logs, queries, and workloads.
- Build and maintain dashboards, alerts, and metrics using Datadog, CloudWatch, or Prometheus.
- Participate in incident response, diagnose pipeline outages, and contribute to root-cause analysis and preventative actions.
- Partner with data engineering, BI, analytics, and platform teams to deliver data accurately and on time.
- Document best practices, contribute to runbooks, and mentor junior engineers.
Requirements
- At least 4 years of experience in data engineering or big data platform operations.
- Production experience with Databricks, Spark, Flink, or Airflow/MWAA.
- Strong SQL skills and experience developing ETL/ELT pipelines.
- Familiarity with AWS services including S3, Glue, Athena, EMR, and EKS.
- Working knowledge of Datadog, CloudWatch, or Prometheus.
- Proficiency in Python, Scala, or Java for pipeline development and automation.
- Familiarity with Terraform or similar infrastructure-as-code tools.
- Strong collaboration and problem-solving skills, including experience working in distributed teams.
- Preferred: production incident response and RCA experience, data governance and access control knowledge, cost optimization experience, and familiarity with GitLab, Jenkins, or GitHub Actions.
- Preferred: Databricks Data Engineer Associate or AWS Solutions Architect – Associate certification.
Tech Stack
Categories
Data Engineering
About Deloitte Advisory
Deloitte Advisory provides consulting and managed services that help organizations manage risk, strengthen controls, investigate fraud, meet regulatory requirements, and execute transactions using analytics and industry expertise. Offerings span cybersecurity, M&A due diligence, finance and regulatory transformation, internal audit, and tech-enabled platforms. It is part of Deloitte’s global network (founded in 1845 and privately held), with U.S. member-firm teams serving corporations and government agencies across sectors.