EY - GDS Consulting - AI And DATA -AI Data Platform Engineer - Databricks- Senior
Ernst and Young28 days ago
Hyderābād, IndiaSenior
Responsibilities
- Design and implement scalable ETL/ELT pipelines using PySpark, Spark SQL, Delta Lake, Databricks Workflows, Lakeflow, and Delta Live Tables.
- Develop reusable ingestion frameworks for batch, streaming, event-driven, CDC, file-based, and API-based processing.
- Build curated, analytics-ready, and AI-ready data products with quality, lineage, semantic context, and operational controls.
- Create reusable platform accelerators for workspace onboarding, pipeline templates, deployment, logging, monitoring, and support operations.
- Implement Git connectivity, branching, pull requests, code reviews, deployment bundles, CI/CD, and controlled environment promotion.
- Integrate Databricks with enterprise APIs, source systems, orchestration platforms, governance tools, security services, and analytics consumers.
- Configure Databricks Genie Spaces, AI/BI dashboards, governed natural-language analytics, Mosaic AI, MLflow, Vector Search, RAG, GraphRAG, and agentic workflows.
- Implement Unity Catalog governance, including access controls, lineage, audit logging, masking, privacy controls, policy enforcement, and AI governance.
- Build Data SRE capabilities for observability, incident management, restartability, SLA/SLO tracking, root-cause analysis, FinOps, and production readiness.
- Mentor engineers and collaborate with architects, security teams, and SRE teams on reusable, secure, production-ready platform services.
Requirements
- 5–10 years of experience in data engineering, data platform operations, analytics engineering, platform engineering, or AI platform enablement.
- Strong hands-on experience with cloud data platforms, APIs, Git connectivity, CI/CD, governed access patterns, SRE practices, and production operations.
- Experience with Databricks Lakehouse, Delta Lake, Lakeflow, Delta Live Tables, Databricks Workflows, Databricks SQL, and Unity Catalog.
- Experience with Python, PySpark, SQL, APIs, Git, testing, Databricks Asset Bundles, and cloud or DevOps technologies.
- Experience with governance, security, lineage, audit, masking, data quality, observability, Data SRE, FinOps, responsible AI, or AI security practices.
- Preferred certifications related to cloud or platform technologies, data engineering, DevOps, security, governance, or AI/ML engineering.
- Architecture awareness, delivery ownership, platform engineering judgment, and the ability to turn standards into reusable frameworks and operational controls.
- Ability to mentor engineers and collaborate with architecture, security, and SRE teams.
Benefits
- Support, coaching, and feedback from experienced colleagues.
- Opportunities to develop new skills and progress your career.
- Individual progression planning, education, coaching, and practical project experience.
- Freedom and flexibility to handle the role in a way that is right for you.
- Interdisciplinary work emphasizing quality and knowledge exchange.
Tech Stack
AWSAzureDatabricksDockerGitGitHub ActionsGoogle Cloud PlatformJenkinsKubernetesMLflowPythonSQLTerraform
Categories
Data EngineeringDevOps
About Ernst and Young
Ernst & Young (EY) provides audit/assurance, tax, consulting, strategy and transactions services to enterprises, financial institutions, and public‑sector clients. Structured as a global network of partner‑owned member firms, it sells professional services on a fee basis, including a dedicated Financial Services Organization for banking, insurance, and capital markets. Headquartered in London, EY was formed in 1989 from the merger of Ernst & Whinney and Arthur Young, and operates in 150+ countries.