Manager | Data Engineer | Chennai | Regulatory & Financial Risk
Deloitte Advisory2 months ago
Chennai, IndiaStaff+
Responsibilities
- Design, develop, and optimize large-scale data pipelines using PySpark.
- Build and maintain ETL/ELT workflows for structured and unstructured data.
- Work with Hadoop, Spark, Impala, Hive, and relational databases.
- Perform data modeling, transformation, pipeline orchestration, and Spark performance optimization.
- Participate in end-to-end SDLC activities, including requirements analysis, design, development, testing, and deployment.
- Collaborate with data scientists, analysts, product teams, and other cross-functional stakeholders.
- Ensure code quality through best practices, code reviews, and version control.
- Troubleshoot data processing issues and production incidents.
- Contribute to automation, monitoring, and logging frameworks.
Requirements
- Strong hands-on experience with Python and PySpark.
- At least 5 years of hands-on PySpark development experience, with the stated preferred range of 5–8 years.
- Strong SQL skills, including advanced querying and performance tuning.
- Experience with distributed data systems, Hadoop, Spark, Impala, Hive, and SQL Server or other relational databases.
- Understanding of data structures, algorithms, performance optimization, ETL pipeline development, and data warehousing.
- Experience working with large datasets in production environments.
- Knowledge of cloud platforms such as Azure, AWS, or GCP.
- Exposure to Airflow or Oozie and experience with CI/CD pipelines.
- Understanding of data lakes, data governance, and data quality frameworks.
- Familiarity with Git and SDLC methodologies; Agile/Scrum experience is preferred.
- Knowledge of C# or other programming languages.
- Strong analytical, problem-solving, communication, collaboration, independent-working, and attention-to-detail skills.
- Bachelor’s degree in any specialization.
Tech Stack
Apache AirflowApache HadoopApache HiveApache SparkAWSAzureC#GitGoogle Cloud PlatformMicrosoft SQL ServerPythonSQL
Categories
Data Engineering
About Deloitte Advisory
Deloitte Advisory provides consulting and managed services that help organizations manage risk, strengthen controls, investigate fraud, meet regulatory requirements, and execute transactions using analytics and industry expertise. Offerings span cybersecurity, M&A due diligence, finance and regulatory transformation, internal audit, and tech-enabled platforms. It is part of Deloitte’s global network (founded in 1845 and privately held), with U.S. member-firm teams serving corporations and government agencies across sectors.