1 month ago
Kolkāta, India +3 moreMid Level
Responsibilities
- Build, maintain, and optimize scalable data pipelines for large-scale data processing.
- Develop ETL/ELT processes using PySpark, Spark, and related tools.
- Assist with designing and deploying data engineering solutions on AWS, Azure, or GCP.
- Support big data processing frameworks and data lakes for structured and unstructured data.
- Collaborate with data scientists, analysts, architects, and engineers to ensure data accuracy and availability.
- Implement secure and efficient data ingestion across storage solutions.
- Develop batch and real-time streaming pipelines for high-volume workloads.
- Create reusable code for data extraction, transformation, and loading.
- Tune Spark jobs and pipelines for scalability and performance.
- Support data governance and security practices across cloud platforms.
Requirements
- At least 3 years of experience is required.
- 15 years of full-time education is required.
- Hands-on experience with the Databricks Unified Data Analytics Platform is required.
- Experience with AWS, Azure, or GCP for data engineering workflows.
- Strong proficiency with PySpark, Spark, or similar scalable data-processing frameworks.
- Understanding of big data architectures, storage, and processing concepts.
- Familiarity with S3, Blob Storage, BigQuery, or Redshift.
- Experience with Apache Airflow or similar data orchestration tools.
- Knowledge of Parquet, Avro, or JSON data formats.
- Strong Python coding skills for building data pipelines.
- Good understanding of SQL and database technologies.
- Strong troubleshooting, debugging, and performance optimization skills.
- Experience with real-time data processing and Kafka or similar tools is preferred.
- Exposure to CI/CD pipelines and DevOps practices in cloud environments is preferred.
- Familiarity with dbt is preferred.
- Experience with Informatica, Talend, or Matillion is preferred.
Tech Stack
Amazon RedshiftApache AirflowApache KafkaApache SparkAWSAzureDatabricksdbtGoogle BigQueryGoogle Cloud PlatformInformaticaPythonSQLTalend
Categories
Data Engineering
About Accenture
Accenture is a global professional services firm providing management consulting, systems integration and technology, cybersecurity, and business process outsourcing for enterprises and governments. It operates a services-driven model delivering projects and managed services, often with major cloud and software partners, across industries worldwide. Headquartered in Dublin and publicly traded on the NYSE (ACN), it originated as Andersen Consulting and adopted the Accenture name in 2001.