9 days ago
Pune, IndiaSenior
Responsibilities
- Design, develop, and maintain scalable Big Data applications and data pipelines.
- Develop high-performance data-processing solutions using Scala and Apache Spark.
- Build and optimize ETL/ELT pipelines for large volumes of structured and unstructured data.
- Work with Hadoop, HDFS, Hive, Kafka, and Spark SQL.
- Tune and optimize Spark jobs and troubleshoot production issues in data pipelines and distributed applications.
- Develop reusable data-processing frameworks and implement data-quality checks, monitoring, and error handling.
- Collaborate with data architects, analysts, business teams, and engineering teams.
- Follow software engineering practices including Git, code reviews, testing, and CI/CD.
Requirements
- Strong hands-on programming experience in Scala and Apache Spark/Spark SQL.
- Strong understanding of Big Data and distributed-computing concepts.
- Experience with Hadoop ecosystem technologies, including Hadoop, HDFS, and Hive.
- Experience building large-scale ETL and data-engineering pipelines.
- Strong SQL skills and experience with Kafka or similar messaging and streaming technologies.
- Understanding of Parquet, Avro, and JSON data formats.
- Experience optimizing and troubleshooting Spark applications.
- Understanding of software development lifecycle and version-control systems.
- Preferred experience with AWS, Azure, or GCP.
- Preferred knowledge of Databricks, Airflow, Delta Lake, Docker, Kubernetes, and CI/CD pipelines.
- Experience in Agile/Scrum environments is preferred.
Benefits
- Work on engaging transformation projects with major international and local banks, insurance companies, payment service providers, and other clients.
- Open, tolerant, diverse, and inclusive workplace culture that values creativity and different perspectives.
- Career advancement opportunities without a forced hierarchy.
- Opportunity to contribute to disruptive work across financial services and energy sectors.
Tech Stack
Apache AirflowApache HadoopApache HiveApache KafkaApache SparkAWSAzureDatabricksDockerGitGoogle Cloud PlatformKubernetesScalaSQL
Categories
Data Engineering
About Capco
Capco is a global technology and management consultancy focused on financial services, helping banks, capital markets, insurance, and wealth firms with digital transformation, risk and compliance, and data/AI-enabled modernization. Founded in 1998 and headquartered in London, it operates as a Wipro company. Its business model centers on project-based consulting, managed services, and solution delivery across strategy, engineering, and platforms for regulated institutions.