Senior Serverless Spark Migration Engineer
Virtasant Inc.4 days ago
Remote, Mexico or Remote, BrazilSenior
Responsibilities
- Lead migrations of enterprise Apache Spark workloads from on-premise Hadoop and Spark environments to AWS and GCP.
- Assess Spark applications, clusters, configurations, dependencies, data flows, and resource utilization.
- Select migration approaches including rehost, replatform, refactor, modernize, or retire.
- Design and implement serverless Spark architectures using AWS EMR Serverless, Amazon S3, AWS Glue, AWS Lake Formation, Google Cloud Dataproc Serverless, Google Cloud Storage, and BigQuery.
- Refactor PySpark, Scala, and Spark SQL applications for cloud portability, scalability, and reliability.
- Troubleshoot and optimize Spark workloads, including partitioning, shuffles, joins, data skew, execution plans, executor configuration, serialization, and SQL execution.
- Benchmark workloads and optimize performance, reliability, and cloud cost.
- Build reusable migration tooling, automation, templates, and frameworks.
- Implement CI/CD and Infrastructure as Code using Terraform.
- Define testing, validation, cutover, rollback, observability, and production-readiness patterns.
- Partner with Data Engineering, ML, Cloud Architecture, Platform Engineering, DevOps/SRE, Security, Governance, and FinOps teams.
Requirements
- 8+ years of experience in data engineering, distributed systems, cloud engineering, or platform engineering.
- 5+ years of hands-on Apache Spark experience in enterprise environments.
- Strong PySpark and/or Scala development experience.
- Proven experience migrating large-scale Spark workloads between infrastructure platforms.
- Hands-on experience with both AWS and GCP.
- Experience with on-premise Hadoop and Spark ecosystems, including HDFS, YARN, and Hive.
- Deep understanding of Spark internals and distributed processing.
- Strong SQL and data engineering fundamentals.
- Experience with cloud data lakes and object storage.
- Strong production troubleshooting and performance-tuning experience.
- Experience with CI/CD, Git, and Infrastructure as Code.
- Ability to own migration work end-to-end, from discovery and architecture through production cutover and optimization.
- Experience with EMR or EMR Serverless, Dataproc or Dataproc Serverless, Glue, Lake Formation, BigQuery, Delta Lake, Iceberg, Kafka, Airflow, Terraform, Docker, or Kubernetes is valuable.
Benefits
- Remote work from Brazil or Mexico.
- Coverage aligned to Pacific hours, 8:00 AM–5:00 PM PST.
Tech Stack
Apache AirflowApache HadoopApache HiveApache KafkaApache SparkAWSDockerGitGoogle BigQueryGoogle CloudGoogle Cloud PlatformKubernetesScalaSQLTerraformYarn
Categories
Data Engineering
About Virtasant Inc.
Virtasant provides cloud and software engineering services to enterprises, focusing on cloud migration and modernization, cost optimization, and AI-enabled transformation. The privately held, fully remote company also develops Helias, a cloud cost optimization platform used to understand, govern, and reduce spend at scale. Founded in 2007 and headquartered in Austin, Texas, it serves global customers across AWS, Azure, and Google Cloud.