4 days ago
Noida, IndiaMid Level / Senior
Responsibilities
- Design, build, and maintain high-volume ETL/ELT pipelines across Hadoop and AWS.
- Develop distributed data processing solutions with PySpark, Spark SQL, Scala, and Spark optimization techniques.
- Implement reusable batch data ingestion frameworks and orchestrate workflows using Airflow, Control-M, Step Functions, or event-driven triggers.
- Build serverless data architectures integrating Glue, Lambda, S3, Redshift, and other AWS services.
- Optimize data workflows through partitioning, bucketing, compression, and Parquet/ORC file formats.
- Design hybrid S3 and HDFS data lake architectures with appropriate governance and compliance controls.
- Define technical specifications, make scalable architecture decisions, and build secure software solutions.
- Apply unit testing, automation, code reviews, and secure coding practices to improve maintainability.
- Monitor and troubleshoot Spark performance issues, job failures, cluster bottlenecks, and production incidents.
- Understand and create data flow diagrams, data models, data lineage, and end-to-end data flows.
- Collaborate with business stakeholders to gather requirements and deliver high-quality datasets.
- Provide guidance to peers and junior team members and collaborate with QA and cross-functional teams on releases.
- Support BAU and year-end activities as needed.
Requirements
- 4 to 8 years of overall ETL experience.
- Strong expertise in Big Data technologies, including Spark and Cloudera.
- Hands-on experience with AWS data services and serverless data architectures.
- Strong experience with Scala, PySpark, Spark optimization, HiveQL, and distributed computing.
- Good understanding of the Hadoop ecosystem, including HDFS, Hive, Spark, YARN, and Kafka.
- Strong SQL experience with Hive and Impala.
- Proficiency in at least one scripting or programming language, including Python or Shell scripting.
- Experience with GitHub and Git commands.
- Expertise in ETL, data warehousing, cloud concepts, data modeling, partitioning strategies, and schema evolution.
- Experience with Airflow, Control-M, or other workflow orchestrators.
- Familiarity with serverless patterns and containerization using Docker, ECS, or EKS.
- Understanding of cloud security and compliance considerations.
- Strong logical, analytical, problem-solving, communication, and stakeholder coordination skills.
- AWS Data Engineer or Developer certification is preferred.
- Ability to work effectively with cross-functional teams and provide peer guidance.
Benefits
- Health benefits begin on Day One.
- Benefits include wellbeing programs, retirement plans, career growth opportunities, continuing education, and training.
- The role may require occasional travel to a physical office location.
- Virtual interviews are conducted by video and may require a camera to be on.
- Employment is contingent on successfully completing applicable background checks.
- Applicants must have work authorization in the employing country that does not now or in the future require visa sponsorship.
Tech Stack
Amazon RedshiftApache AirflowApache HadoopApache HiveApache KafkaApache SparkAWSDockerGitPythonScalaSQLYarn
Categories
Data Engineering
About Alight
Alight provides cloud-based benefits administration and human capital solutions to large employers, combining its Alight Worklife platform with outsourced services for health, wealth, leave, and wellbeing programs. It earns revenue through SaaS subscriptions and long-term managed services and consulting. Headquartered in Deerfield, Illinois, Alight was carved out of Aon’s benefits outsourcing business by Blackstone in 2017 and is now publicly traded on the NYSE (ticker: ALIT).