Manager | Data Engineer | Chennai | Regulatory & Financial Risk
Deloitte Advisory2 months ago
Chennai, IndiaStaff+
Responsibilities
- Design, develop, and optimize large-scale data pipelines using PySpark.
- Build and maintain ETL/ELT workflows for structured and unstructured data.
- Work with Hadoop, Spark, Impala, Hive, and relational databases.
- Optimize Spark job performance, memory usage, and execution time.
- Perform data modeling, transformation, and pipeline orchestration.
- Participate in requirements analysis, design, development, testing, and deployment across the SDLC.
- Troubleshoot data-processing issues and production incidents.
- Contribute to automation, monitoring, and logging frameworks.
- Ensure code quality through best practices, code reviews, and version control.
- Collaborate with data scientists, analysts, product teams, and other cross-functional stakeholders.
Requirements
- Strong hands-on experience with Python and PySpark is mandatory.
- At least 5 years of experience, with the preferred range being 5–8 years of hands-on PySpark development and real-time project delivery.
- Strong SQL skills, including advanced querying and performance tuning.
- Experience with distributed data systems, ETL pipeline development, data warehousing, and large datasets in production.
- Experience with Impala, Hive, SQL Server, or other relational databases.
- Strong understanding of data structures, algorithms, and performance optimization.
- Knowledge of cloud platforms such as Azure, AWS, or GCP.
- Exposure to Airflow or Oozie and experience with CI/CD pipelines.
- Familiarity with Git and SDLC methodologies; Agile/Scrum is preferred.
- Knowledge of data lakes, data governance, and data quality frameworks.
- Knowledge of C# or other programming languages.
- Bachelor’s degree in any specialization.
- Strong analytical, problem-solving, communication, collaboration, independent-work, and attention-to-detail abilities.
Tech Stack
Apache AirflowApache HadoopApache HiveApache SparkAWSAzureC#GitGoogle Cloud PlatformMicrosoft SQL ServerPythonSQL
Categories
Data Engineering
About Deloitte Advisory
Deloitte Advisory provides consulting and managed services that help organizations manage risk, strengthen controls, investigate fraud, meet regulatory requirements, and execute transactions using analytics and industry expertise. Offerings span cybersecurity, M&A due diligence, finance and regulatory transformation, internal audit, and tech-enabled platforms. It is part of Deloitte’s global network (founded in 1845 and privately held), with U.S. member-firm teams serving corporations and government agencies across sectors.