2 months ago
Kolkāta, India or Bengaluru, IndiaSenior
Responsibilities
- Build, maintain, and optimize scalable data pipelines for large-scale data processing.
- Develop ETL/ELT processes using PySpark, Spark, and related tools.
- Design and deploy data engineering solutions on AWS, Azure, or GCP.
- Support Big Data processing frameworks and data lakes for structured and unstructured data.
- Implement secure and efficient data ingestion across storage solutions.
- Develop batch and real-time streaming pipelines for high-volume workloads.
- Maintain reusable code for data extraction, transformation, and loading.
- Tune Spark jobs and data pipelines for scalability and efficiency.
- Support data governance and security practices across cloud platforms.
- Collaborate with data scientists, analysts, architects, and engineers to ensure data accuracy and availability.
Requirements
- At least 5 years of professional experience.
- At least 6 years of experience with the Databricks Unified Data Analytics Platform.
- Experience with AWS, Azure, or GCP for data engineering workflows.
- Strong proficiency with PySpark, Spark, or similar scalable data processing frameworks.
- Strong Python coding skills for data pipeline development.
- Understanding of Big Data architectures, data storage, and data processing.
- Familiarity with S3, Blob Storage, BigQuery, or Redshift.
- Experience with Apache Airflow or similar data orchestration tools.
- Knowledge of Parquet, Avro, or JSON data formats.
- Good understanding of SQL and database technologies.
- Experience with real-time data processing and Kafka or similar tools.
- Exposure to CI/CD pipelines and DevOps practices in cloud environments.
- Familiarity with dbt for data transformation workflows.
- Experience with Informatica, Talend, or Matillion.
- Excellent troubleshooting, debugging, and performance optimization skills.
- Completion of 15 years of full-time education.
Tech Stack
Amazon RedshiftApache AirflowApache KafkaApache SparkAWSAzureDatabricksdbtGoogle BigQueryGoogle Cloud PlatformInformaticaPythonSQLTalend
Categories
Data Engineering
About Accenture
Accenture is a global professional services firm providing management consulting, systems integration and technology, cybersecurity, and business process outsourcing for enterprises and governments. It operates a services-driven model delivering projects and managed services, often with major cloud and software partners, across industries worldwide. Headquartered in Dublin and publicly traded on the NYSE (ACN), it originated as Andersen Consulting and adopted the Accenture name in 2001.