Capco

AWS Data Engineer

Capco
Apply
1 day ago
Bengaluru, IndiaSenior / Staff+

Responsibilities

  • Design, develop, test, maintain, and optimize scalable Databricks data pipelines and transformation workflows.
  • Build batch and streaming data processing solutions using Apache Spark, PySpark, SQL, and Delta Lake.
  • Design cloud-based lakehouse platforms, data models, storage strategies, partitioning, and lifecycle management processes.
  • Develop data ingestion, integration, orchestration, validation, reconciliation, monitoring, error handling, and recovery workflows.
  • Integrate structured, semi-structured, and unstructured sources with enterprise databases, APIs, cloud storage, and analytical platforms.
  • Develop CI/CD pipelines, automated tests, reusable frameworks, and deployment processes for Databricks code and configurations.
  • Implement data governance, access controls, security configurations, metadata management, lineage, monitoring, and operational resilience.
  • Troubleshoot Spark and Databricks performance issues and resolve defects across development, testing, and production environments.
  • Engage client stakeholders, contribute to architecture and technical design decisions, and own complex technical workstreams.
  • Provide technical guidance, code reviews, mentoring, delivery coordination, estimation, sprint planning, and stakeholder communication.

Requirements

  • Typically 6+ years of experience in data engineering, software engineering, or related technology roles, including substantial hands-on Databricks experience.
  • Advanced proficiency in Python, PySpark, and SQL, with strong knowledge of Apache Spark architecture and distributed data processing.
  • Hands-on experience with Delta Lake, ETL/ELT pipelines, lakehouse architecture, data modeling, data warehousing, and Spark performance optimization.
  • Strong AWS development experience with several of Lambda, Glue, Step Functions, IAM roles, Lake Formation, EventBridge, SNS, SQS, EC2, Security Groups, CloudFormation, RDS, DynamoDB, and Redshift.
  • Experience with Databricks Workflows or Azure Data Factory, Git, CI/CD pipelines, data quality checks, logging, monitoring, and data governance.
  • Ability to contribute to technical design and solution architecture, independently deliver complex workstreams, communicate with technical and non-technical stakeholders, and mentor junior team members.
  • Experience working in Agile software delivery environments and managing delivery risks, dependencies, competing priorities, and client relationships.
  • Desirable qualifications include financial services experience, Databricks Data Engineer Associate or Professional certification, Unity Catalog, Delta Live Tables or Lakeflow Spark Declarative Pipelines, Databricks SQL, Spark Structured Streaming, Kafka, Terraform, Microsoft Fabric, Snowflake, Synapse Analytics, cloud migration, Data Vault, dimensional modeling, data observability, metadata management, lineage, and cost optimization experience.
  • Excellent verbal communication, interpersonal, organizational, analytical, problem-solving, collaborative, and client-focused skills.

Benefits

  • Projects with major international and local banks, insurers, payment providers, and other financial services clients.
  • Open, diverse, inclusive, and creative culture with opportunities for career growth and no forced hierarchy.
  • Role is based in Bangalore or Pune.

Tech Stack

Amazon DynamoDBAmazon RedshiftApache KafkaApache SparkAWSDatabricksGitSnowflakeSQLTerraform

Categories

Data Engineering
Capco

About Capco

5,001-10,000 employees

Capco is a global technology and management consultancy focused on financial services, helping banks, capital markets, insurance, and wealth firms with digital transformation, risk and compliance, and data/AI-enabled modernization. Founded in 1998 and headquartered in London, it operates as a Wipro company. Its business model centers on project-based consulting, managed services, and solution delivery across strategy, engineering, and platforms for regulated institutions.

Contact me