Staff Data Platform Engineer (AWS, Databricks, Azure, IaC, Solution Architect, Java/Python, 6 - 10 Years)
Visa8 days ago
Bengaluru, IndiaStaff+
Responsibilities
- Design, develop, maintain, and operate scalable cloud data platform services across AWS, Azure, and Databricks.
- Build reusable platform capabilities, self-service onboarding solutions, engineering frameworks, and data integration pipelines.
- Migrate and modernize analytical, data, and AI/ML workloads from Hadoop-based platforms to cloud environments.
- Automate infrastructure provisioning, platform deployment, and operational processes using Terraform and CloudFormation.
- Implement data governance, metadata management, lineage, security, and access-management capabilities.
- Monitor and optimize platform performance, scalability, reliability, operational efficiency, and cost.
- Partner with architecture, security, product, engineering, and application teams to onboard workloads and enable cloud adoption.
- Troubleshoot complex production issues and improve platform reliability and operational excellence.
- Provide technical guidance and mentorship while promoting engineering best practices.
Requirements
- At least 8 years of experience in cloud data engineering, big data engineering, platform engineering, or data platform engineering.
- Strong hands-on experience building and operating enterprise-scale cloud data platforms.
- Expertise with AWS and Azure data platform services, including AWS EMR, EMR on EKS, Glue, Athena, S3, IAM, Lake Formation, ADLS, Azure Databricks, Synapse Analytics, Data Factory, and Azure Kubernetes Service.
- Hands-on experience with Databricks, Delta Lake, Unity Catalog, and Lakehouse architecture patterns.
- Experience migrating workloads from Hadoop-based platforms to cloud environments.
- Strong proficiency in Python, Scala, or Java.
- Hands-on experience with Terraform or CloudFormation and strong expertise in Spark, Hive, Hadoop, and distributed data processing frameworks.
- Understanding of data governance, security, compliance, metadata management, and access controls, along with strong Linux, operating system, networking, and troubleshooting skills.
- Preferred qualifications include multi-tenant data platform experience, Kubernetes and containerized workload experience, observability and reliability engineering experience, cloud cost optimization experience, AI/ML platform experience, mentoring experience, and large-scale cloud transformation experience.
Tech Stack
Categories
Data EngineeringDevOps
About Visa
Visa builds and operates a global digital payments network that connects consumers, merchants, financial institutions, and governments. It earns fees for processing card transactions and offering risk, data, and value‑added services through VisaNet and products like tokenization and tap‑to‑pay. Founded in 1958 and headquartered in Foster City, California, Visa is a public company (NYSE: V) serving clients in more than 200 countries and territories.