EY - GDS Consulting - AI And DATA -AWS Data Engineer - Manager
Ernst and Young21 days ago
Hyderābād, IndiaStaff+
Responsibilities
- Design and build production-grade AWS data pipelines using Amazon S3, AWS Glue, PySpark, Athena, Redshift, EMR, MWAA/Airflow, Step Functions, Lambda, EventBridge, CloudWatch, IAM, and KMS.
- Develop reusable ingestion frameworks for batch, streaming, event-driven, CDC, API-based, file-based, database, and third-party service integration patterns.
- Build curated raw, standardized, trusted, and consumption data layers using lakehouse design, partitioning, metadata management, and file-format optimization.
- Optimize Spark, Glue, and EMR workloads for performance, cost efficiency, scalability, and operational stability.
- Create reusable AWS platform accelerators for onboarding, pipeline templates, orchestration, monitoring, reconciliation, deployment, logging, and support runbooks.
- Implement Git connectivity, branching, pull requests, code reviews, CI/CD, infrastructure as code, controlled releases, and environment promotion.
- Integrate AWS data platforms with enterprise APIs, source applications, messaging and event services, governance tools, security platforms, and analytics consumers.
- Integrate AWS data platforms with SageMaker for data preparation, feature engineering, model training, deployment, MLOps workflows, and inference-ready data products.
- Support SageMaker Pipelines, Feature Store, Model Registry, Model Monitor, Bedrock where relevant, vector stores, semantic search, and RAG-ready data products.
- Implement data security, governed access, observability, SLA/SLO tracking, alerting, retry logic, restartability, root-cause analysis, incident response, and production reliability management.
- Mentor engineers and collaborate with architecture, infrastructure, IAM, network, DBA, application, security, SRE, and support teams.
Requirements
- 8–11 years of experience in data engineering, data platform operations, analytics engineering, platform engineering, or AI platform enablement.
- Strong hands-on implementation experience with cloud data platforms, APIs, Git connectivity, CI/CD, governed access patterns, SRE practices, and production operations.
- Experience with AWS Glue, Amazon S3, Athena, Redshift, EMR, MWAA/Airflow, Step Functions, Lambda, and EventBridge.
- Experience with Python, PySpark, SQL, Git, shell scripting, unit testing, integration testing, and data pipeline testing.
- Experience with Terraform/OpenTofu, CloudFormation, GitHub Actions, Azure DevOps, Jenkins, Docker, Kubernetes/EKS, and policy-as-code.
- Experience with IAM, KMS, Lake Formation, Glue Data Catalog, Macie, Immuta, Purview, CloudTrail, data quality, observability, Data SRE, and FinOps practices.
- Strong hands-on engineering ability with architecture awareness, delivery ownership, reusable framework development, secure implementation patterns, operational controls, and production-ready services.
- Preferred certifications aligned with cloud/platform, data engineering, DevOps, security, governance, and AI/ML engineering domains.
- Ability to mentor engineers, collaborate with architects and security/SRE teams, and adopt AI-native and agentic engineering methods.
Benefits
- Support, coaching, and feedback from engaging colleagues.
- Opportunities to develop new skills and progress your career.
- Individual progression planning, education, coaching, and practical experience.
- Freedom and flexibility to handle the role in a way that is right for you.
- Work on varied and meaningful projects for clients ranging from start-ups to Fortune 500 companies in an interdisciplinary environment.
Tech Stack
Amazon RedshiftApache AirflowApache SparkAWSDockerGitGitHub ActionsJenkinsKubernetesPythonSQLTerraform
Categories
Data Engineering
About Ernst and Young
Ernst & Young (EY) provides audit/assurance, tax, consulting, strategy and transactions services to enterprises, financial institutions, and public‑sector clients. Structured as a global network of partner‑owned member firms, it sells professional services on a fee basis, including a dedicated Financial Services Organization for banking, insurance, and capital markets. Headquartered in London, EY was formed in 1989 from the merger of Ernst & Whinney and Arthur Young, and operates in 150+ countries.