6 days ago
Chennai, India
Mid Level / Senior / Staff+
H1B Sponsor
Responsibilities
- Lead the design and implementation of data pipelines for ingesting, processing, and aggregating large datasets on AWS.
- Develop high-quality, maintainable code primarily using Java and Python, adhering to software development best practices.
- Utilize Infrastructure as Code (Terraform) for provisioning and managing all AWS resources.
- Develop and optimize data transformation jobs using AWS Glue and Apache Spark.
- Design and implement efficient data storage solutions utilizing Amazon S3 and Parquet file formats.
- Develop and deploy serverless functions (AWS Lambdas) for various data processing tasks and API logic.
- Configure and manage API Gateways to expose data securely and efficiently.
- Implement and maintain robust AWS networking configurations.
- Ensure high standards for logging, monitoring, and alerting across the entire data platform.
- Implement and enforce secure authentication mechanisms for API access and data access.
- Monitor and optimize cloud costs, implementing strategies for cost tracking and alerting.
- Collaborate with cross-functional teams to understand data requirements and deliver effective solutions.
- Provide technical leadership and mentorship to other team members.
Requirements
- Proven experience as a Lead Engineer, with a strong focus on AWS.
- Experience with Java and/or Python programming languages.
- Extensive experience with large-scale data processing and aggregation.
- Deep expertise in AWS services such as S3, Lambda, Glue, and EMR.
- Strong proficiency in Infrastructure as Code (Terraform).
- Experience with logging and monitoring solutions like CloudWatch and CloudTrail.
- Solid understanding of authentication and authorization mechanisms.
- Demonstrated ability to track, analyze, and optimize AWS cloud costs.
- Excellent problem-solving and communication skills.
- Ability to work independently and as part of a team in a fast-paced environment.
Benefits
- Comprehensive benefits package supporting health, wellbeing, and financial future.
- Health coverage and Employee Assistance Program (EAP) with free mental health support.
- Life and disability insurance, Employee Stock Purchase Program (ESPP), and retirement/pension plan.
- Wellness dollars, tuition reimbursement, and family-forming benefits.
Tech Stack
Apache SparkAWSJavaPythonTerraform
Categories
BackendData EngineeringDevOps