Responsibilities
- Design and implement scalable, secure, and cost-optimized AWS data architectures.
- Develop and maintain ETL pipelines using AWS Lambda and AWS Glue ETL.
- Configure AWS Glue Crawlers and the Glue Data Catalog, including schema evolution.
- Build, optimize, and unit test Apache Spark applications using PySpark.
- Design and optimize Apache Iceberg data lakes, including table compaction and performance tuning.
- Work with Avro, Parquet, JSON, XML, and CSV data formats.
- Orchestrate event-driven workflows using AWS Step Functions and Amazon EventBridge.
- Integrate Starburst with Lambda and Glue ETL jobs for federated querying.
- Implement automated testing and deployment pipelines.
- Perform unit testing and performance tuning of Spark and Python applications.
Requirements
- Strong understanding of AWS architecture best practices, scalability, security, and cost optimization.
- Hands-on experience with AWS Lambda, Glue ETL, Athena, S3, DynamoDB, Step Functions, EventBridge, SNS, and SQS.
- Deep experience developing, testing, and optimizing Apache Spark applications with PySpark and Scala.
- Strong Python programming skills with pandas, requests, json, and awswrangler.
- Experience with Apache Kafka and Confluent Kafka.
- Experience designing and optimizing Apache Iceberg data lakes, including compaction and optimization techniques.
- Experience with real-time streaming, event-driven architectures, and API-driven data applications.
Tech Stack
Categories
About EXL
Choosing a digital partner is about more than capabilities — it’s about collaboration and character. Unrealistic overhauls and off-the-shelf products ignore what matters most — your unique needs, culture, goals, and your legacy data and technology environments. At EXL, our collaboration is built on ongoing listening and learning to adapt our methodologies. We’re your business evolution partner—tailoring solutions that make the most of data to make better business decisions and drive more intelligence into your increasingly digital operations. Whether your goals are scaling the use of AI and digital, redesign operating models, or driving better and faster decisions, we’re here to partner with you to help you gain—and maintain—competitive advantage with efficient, sustainable models at scale. Our expertise in transformation, data science, and change management helps make your business more efficient and effective, improve customer relationships and enhance revenue growth. Instead of focusing on multi-year, resource- and time-intensive platform designs or migrations, we look deeper at your entire value chain to integrate strategies with impact. We use our specialization in analytics, digital interventions, and operations management—alongside deep industry expertise — to deliver solutions that help you outperform the competition. At EXL, it’s all about outcomes—your outcomes—and delivering success on your terms. Share your goals with us and together, we’ll optimize how you leverage data to drive your business forward. For more information, visit www.exlservice.com.