EY - GDS Consulting - AI And DATA -AWS Data Engineer - Manager
Ernst and Young7 days ago
Kolkāta, IndiaStaff+
Responsibilities
- Design and build production-grade AWS data pipelines using S3, Glue, PySpark, Athena, Redshift, EMR, MWAA/Airflow, Step Functions, Lambda, EventBridge, CloudWatch, IAM, and KMS.
- Develop reusable ingestion frameworks for batch, streaming, event-driven, CDC, API-based, file-based, database, and third-party service integration patterns.
- Build curated lakehouse data layers and optimize Spark, Glue, and EMR workloads for performance, cost, scalability, and operational stability.
- Create reusable AWS platform accelerators for onboarding, pipeline templates, orchestration, monitoring, reconciliation, deployment, logging, and support runbooks.
- Implement Git-based delivery, branching, pull requests, code reviews, CI/CD, infrastructure as code, controlled releases, and environment promotion.
- Integrate AWS data platforms with enterprise APIs, source applications, messaging and event services, governance tools, security platforms, and analytics consumers.
- Integrate data platforms with SageMaker for data preparation, feature engineering, model training, deployment, MLOps workflows, and inference-ready data products.
- Support SageMaker Pipelines, Feature Store, Model Registry, Model Monitor, Bedrock where relevant, vector stores, semantic search, and RAG-ready data products.
- Implement data security, governed access, observability, SLA/SLO tracking, alerting, retry logic, restartability, root-cause analysis, incident response, and production reliability management.
- Mentor engineers and collaborate with architects, infrastructure, IAM, network, DBA, application, security, SRE, and support teams.
Requirements
- 8–11 years of experience in data engineering, data platform operations, analytics engineering, platform engineering, or AI platform enablement.
- Strong hands-on implementation experience with cloud data platforms, APIs, Git connectivity, CI/CD, governed access patterns, SRE practices, and production operations.
- Hands-on experience with AWS data platforms including Glue, S3, Athena, Redshift, EMR, MWAA/Airflow, Step Functions, Lambda, EventBridge, and CloudWatch.
- Experience with Python, PySpark, SQL, Git, shell scripting, unit testing, integration testing, and data pipeline testing.
- Experience with Terraform/OpenTofu, CloudFormation, GitHub Actions, Azure DevOps, Jenkins, Docker, Kubernetes/EKS, and policy-as-code.
- Experience with SageMaker, SageMaker Pipelines, Feature Store, Model Registry, Model Monitor, agentic AI, or RAG-ready data products.
- Experience with IAM, KMS, Lake Formation, Glue Data Catalog, Macie, Immuta, Purview, CloudTrail, data quality, observability, Data SRE, and FinOps.
- Preferred certifications aligned with relevant cloud/platform, data engineering, DevOps, security, governance, and AI/ML engineering stacks.
- Architecture awareness, delivery ownership, a platform engineering mindset, and the ability to turn standards into reusable, secure, production-ready frameworks.
- Ability to mentor engineers, collaborate across architecture, security, and SRE teams, and adopt newer AI-native and agentic engineering methods.
Benefits
- Support, coaching, and feedback from engaging colleagues.
- Opportunities to develop new skills and progress your career.
- Individual progression planning, education, coaching, and practical experience.
- Freedom and flexibility to handle the role in a way that is right for you.
- Work in an interdisciplinary environment with varied client projects and knowledge exchange.
Tech Stack
Categories
Data EngineeringDevOps
About Ernst and Young
Ernst & Young (EY) provides audit/assurance, tax, consulting, strategy and transactions services to enterprises, financial institutions, and public‑sector clients. Structured as a global network of partner‑owned member firms, it sells professional services on a fee basis, including a dedicated Financial Services Organization for banking, insurance, and capital markets. Headquartered in London, EY was formed in 1989 from the merger of Ernst & Whinney and Arthur Young, and operates in 150+ countries.