EY - GDS Consulting - AI And DATA -AI Data Platform Engineer - AWS - Senior
Ernst and Young12 days ago
Bengaluru, IndiaSenior
Responsibilities
- Design and build production-grade AWS data pipelines using S3, Glue, PySpark, Athena, Redshift, EMR, MWAA/Airflow, Step Functions, Lambda, EventBridge, CloudWatch, IAM, and KMS.
- Develop reusable ingestion frameworks for batch, streaming, event-driven, CDC, API-based, file-based, database, and third-party service integration patterns.
- Build and optimize curated lakehouse layers, metadata management, partitioning, and file formats for performance, cost efficiency, scalability, and stability.
- Create reusable AWS platform accelerators for onboarding, pipeline templates, orchestration, monitoring, reconciliation, deployment, logging, and support runbooks.
- Implement Git connectivity, branching, pull requests, code reviews, CI/CD, infrastructure as code, controlled releases, and environment promotion.
- Integrate AWS data platforms with enterprise APIs, source applications, messaging and event services, governance tools, security platforms, and downstream analytics consumers.
- Integrate AWS data platforms with SageMaker for data preparation, feature engineering, model training, deployment, MLOps workflows, and inference-ready data products.
- Support SageMaker Pipelines, Feature Store, Model Registry, Model Monitor, Bedrock where relevant, vector stores, semantic search, and RAG-ready data products.
- Implement data security and governance controls using IAM, KMS, Secrets Manager, VPC endpoints, Lake Formation, Glue Data Catalog, Macie, CloudTrail, Immuta, Purview, and related tools.
- Own Data SRE activities including observability, SLA/SLO tracking, alerting, retry logic, restartability, root-cause analysis, incident response, and production reliability management.
- Collaborate with infrastructure, IAM, network, DBA, application, architecture, security, SRE, and support teams, and mentor engineers.
Requirements
- 5–10 years of experience in data engineering, data platform operations, analytics engineering, platform engineering, or AI platform enablement.
- Strong hands-on implementation experience with cloud data platforms, APIs, Git connectivity, CI/CD, governed access patterns, SRE practices, and production operations.
- Hands-on experience with AWS data and platform services including Glue, S3, Athena, Redshift, EMR, MWAA/Airflow, Step Functions, Lambda, EventBridge, and CloudWatch.
- Experience with Python, PySpark, SQL, APIs, Git, shell scripting, unit testing, integration testing, and data pipeline testing.
- Experience with Terraform/OpenTofu, CloudFormation, GitHub Actions, Azure DevOps, Jenkins, Docker, Kubernetes/EKS, and policy-as-code.
- Experience with IAM, KMS, Lake Formation, Glue Data Catalog, Macie, Immuta, Purview, CloudTrail, data quality, observability, Data SRE, and FinOps practices.
- Experience integrating SageMaker capabilities such as Pipelines, Feature Store, Model Registry, and Model Monitor; Bedrock experience is optional.
- Preferred certifications aligned with cloud/platform technologies, data engineering, DevOps, security, governance, and AI/ML engineering.
- Architecture awareness, delivery ownership, a platform engineering mindset, and the ability to create reusable frameworks, secure implementation patterns, operational controls, and production-ready services.
- Ability to mentor engineers, collaborate with architects and security/SRE teams, and adopt AI-native and agentic engineering methods.
Benefits
- Support, coaching, and feedback from colleagues.
- Opportunities to develop new skills and progress your career.
- Individual progression planning, education, coaching, and practical experience.
- Freedom and flexibility to handle the role in a way that suits the employee.
- Meaningful projects, challenging assignments, interdisciplinary collaboration, and knowledge exchange.
Tech Stack
Categories
Data EngineeringDevOps
About Ernst and Young
Ernst & Young (EY) provides audit/assurance, tax, consulting, strategy and transactions services to enterprises, financial institutions, and public‑sector clients. Structured as a global network of partner‑owned member firms, it sells professional services on a fee basis, including a dedicated Financial Services Organization for banking, insurance, and capital markets. Headquartered in London, EY was formed in 1989 from the merger of Ernst & Whinney and Arthur Young, and operates in 150+ countries.