2 months ago
Pune, IndiaSenior
Responsibilities
- Design and implement scalable, secure, and cost-optimized AWS data architectures.
- Develop and maintain ETL pipelines using AWS Lambda and AWS Glue ETL.
- Configure AWS Glue Crawlers and the Glue Data Catalog, including schema evolution.
- Build, optimize, and unit test Apache Spark applications using PySpark.
- Design and optimize Apache Iceberg data lakes, including table compaction and performance tuning.
- Work with Avro, Parquet, JSON, XML, and CSV data formats.
- Orchestrate event-driven workflows using AWS Step Functions and Amazon EventBridge.
- Integrate Starburst with Lambda and Glue ETL jobs for federated querying.
- Implement automated testing and deployment pipelines.
- Perform unit testing and performance tuning of Spark and Python applications.
Requirements
- Strong understanding of AWS architecture best practices, scalability, security, and cost optimization.
- Hands-on experience with AWS Lambda, Glue ETL, Athena, S3, DynamoDB, Step Functions, EventBridge, SNS, and SQS.
- Deep experience developing, testing, and optimizing Apache Spark applications with PySpark and Scala.
- Strong Python programming skills with pandas, requests, json, and awswrangler.
- Experience with Apache Kafka and Confluent Kafka.
- Experience designing and optimizing Apache Iceberg data lakes, including compaction and optimization techniques.
- Experience with real-time streaming, event-driven architectures, and API-driven data applications.
Tech Stack
Categories
Data Engineering
About EXL
EXL provides data analytics, AI, and digital operations services for enterprises in insurance, healthcare, banking, and other regulated industries. It delivers consulting, managed services, and technology platforms to improve decision-making, risk management, and operational efficiency. Founded in 1999 and headquartered in New York, EXL is a public company listed on the NASDAQ (ticker: EXLS).