Lead Senior Associate | Data Engineering | Hyderabad | Engineering as a Service/ Operate
Deloitte Advisory24 days ago
Hyderābād, IndiaSenior
Responsibilities
- Design, maintain, optimize, and operate Spark and Flink pipelines running in Databricks and Airflow.
- Ensure the reliability and availability of production data workflows and troubleshoot notebooks, jobs, and orchestration workflows.
- Tune Spark jobs, clusters, and pipelines for performance and cost efficiency by analyzing logs, queries, and workloads.
- Build and maintain dashboards, alerts, and metrics using Datadog, CloudWatch, or Prometheus.
- Participate in incident response, diagnose pipeline outages, contribute to root-cause analysis, and implement preventative actions.
- Partner with data engineering, BI, analytics, and platform teams to support data delivery and operations.
- Document best practices, contribute to runbooks, and mentor junior engineers.
Requirements
- At least 4 years of experience in data engineering or big data platform operations.
- Hands-on production experience with Databricks, Spark, Flink, or Airflow/MWAA.
- Strong SQL skills and experience developing ETL/ELT pipelines.
- Familiarity with AWS services including S3, Glue, Athena, EMR, and EKS.
- Working knowledge of Datadog, CloudWatch, or Prometheus.
- Proficiency in Python, Scala, or Java for pipeline development and automation.
- Familiarity with Terraform or similar infrastructure-as-code tools.
- Strong collaboration and problem-solving skills in distributed teams.
- Experience with production incident response and root-cause analysis is preferred.
- Familiarity with CI/CD pipelines using GitLab, Jenkins, or GitHub Actions is preferred.
- Exposure to data governance, access control, and cost optimization practices is preferred.
- Databricks Data Engineer Associate or AWS Solutions Architect – Associate certification is preferred.
- Bachelor’s degree in Computer Science, Engineering, or a related technical field, or equivalent experience.
Tech Stack
Categories
Data Engineering
About Deloitte Advisory
Deloitte Advisory provides consulting and managed services that help organizations manage risk, strengthen controls, investigate fraud, meet regulatory requirements, and execute transactions using analytics and industry expertise. Offerings span cybersecurity, M&A due diligence, finance and regulatory transformation, internal audit, and tech-enabled platforms. It is part of Deloitte’s global network (founded in 1845 and privately held), with U.S. member-firm teams serving corporations and government agencies across sectors.