Lead Senior Associate | Data Engineering | Hyderabad | Engineering as a Service/ Operate
Deloitte Advisory24 days ago
Hyderābād, IndiaSenior
Responsibilities
- Design, maintain, and optimize Spark and Flink pipelines running in Databricks and Airflow.
- Ensure the reliability and availability of production data workflows.
- Debug and resolve issues in notebooks, jobs, and orchestration workflows.
- Tune Spark jobs, clusters, and pipelines for performance and cost efficiency.
- Analyze logs, queries, and workloads to identify bottlenecks and implement improvements.
- Build and maintain dashboards, alerts, and metrics using Datadog, CloudWatch, or Prometheus.
- Monitor pipeline health and support incident response, root-cause analysis, and preventative actions.
- Collaborate with data engineering, BI, analytics, and platform teams to support timely data delivery.
- Document best practices, contribute to runbooks, and mentor junior engineers.
Requirements
- At least 4 years of experience in data engineering or big data platform operations.
- Hands-on production experience with Databricks, Spark, Flink, or Airflow (MWAA).
- Strong SQL skills and experience developing ETL/ELT pipelines.
- Familiarity with AWS services including S3, Glue, Athena, EMR, and EKS.
- Working knowledge of Datadog, CloudWatch, or Prometheus.
- Proficiency in Python, Scala, or Java for pipeline development and automation.
- Familiarity with infrastructure-as-code tools such as Terraform.
- Strong collaboration and problem-solving skills, including experience working in distributed teams.
- Experience with production incident response and root-cause analysis is preferred.
- Familiarity with CI/CD pipelines using GitLab, Jenkins, or GitHub Actions is preferred.
- Exposure to data governance, access control, and cost optimization practices is preferred.
- Databricks Data Engineer Associate or AWS Solutions Architect – Associate certification is preferred.
- Bachelor’s degree in Computer Science, Engineering, or a related technical field, or equivalent experience.
Tech Stack
Categories
Data Engineering
About Deloitte Advisory
Deloitte Advisory provides consulting and managed services that help organizations manage risk, strengthen controls, investigate fraud, meet regulatory requirements, and execute transactions using analytics and industry expertise. Offerings span cybersecurity, M&A due diligence, finance and regulatory transformation, internal audit, and tech-enabled platforms. It is part of Deloitte’s global network (founded in 1845 and privately held), with U.S. member-firm teams serving corporations and government agencies across sectors.