Ernst and Young

EY - GDS Consulting - AI And DATA - Databricks Architect - Senior Manager

Ernst and Young
Apply
18 days ago
Kochi, IndiaStaff+

Responsibilities

  • Architect and deliver scalable Data and AI solutions on the Databricks Lakehouse platform across ingestion, transformation, storage, governance, and consumption layers.
  • Design and optimize distributed data processing pipelines using Hadoop, Apache Spark, and PySpark.
  • Build reusable Python frameworks and production-grade batch and streaming data pipelines.
  • Define cloud data architecture on AWS and/or Azure, including storage, IAM, networking, security, monitoring, and Databricks integration.
  • Own ETL/ELT design, data ingestion, transformation, orchestration, and data quality frameworks.
  • Design dimensional data models using star and snowflake schemas, fact and dimension tables, and slowly changing dimensions.
  • Build Agentic AI solutions with LLM integration, tool/function calling, agent workflows, guardrails, and enterprise data integration.
  • Architect and implement Databricks-based RAG solutions covering document ingestion, chunking, embeddings, vector search, retrieval, grounding, and evaluation.
  • Establish Git-based version control, code reviews, and CI/CD practices across development, test, and production environments.
  • Lead performance optimization and production deployment of Databricks jobs, workflows, and pipelines.
  • Participate in RFP/RFI responses, technical proposals, solution shaping, effort estimation, and architecture presentations.
  • Lead customer workshops, translate business requirements into technical architectures, and mentor engineering teams.

Requirements

  • 13+ years of experience in Data Engineering or Data Architecture.
  • Strong hands-on expertise with Databricks Lakehouse, Delta Lake, Databricks SQL, Unity Catalog, Jobs/Workflows, performance optimization, and production deployments.
  • Strong knowledge of the Hadoop ecosystem, HDFS, Hive, Apache Spark, and PySpark.
  • Strong Python proficiency for scalable data pipelines and frameworks.
  • Experience with AWS and/or Azure cloud architecture, including storage, IAM, networking, security, monitoring, and cloud-native integrations.
  • Expertise in ETL/ELT, batch and streaming processing, ingestion, transformation, orchestration, data quality, and scalable pipeline architecture.
  • Strong SQL and dimensional modeling skills, including star/snowflake schemas, fact/dimension models, SCD, and analytical data models.
  • Hands-on experience building Agentic AI and Databricks RAG solutions, including LLMs, tool/function calling, agent workflows, guardrails, embeddings, vector search, retrieval, grounding, and evaluation.
  • Knowledge of Git, branching strategies, pull requests, code reviews, CI/CD pipelines, and multi-environment deployment.
  • Ability to contribute to strategic RFP/RFI responses, solution shaping, technical proposals, effort estimates, architecture presentations, and customer workshops.
  • Strong client-facing communication skills and ability to translate business requirements into scalable technical solutions.
  • Consulting, systems integrator, or client-facing delivery experience is preferred.
  • Relevant Databricks or cloud certifications, such as Databricks Data Engineer, Databricks Solutions Architect, AWS, or Azure certifications, are preferred.
  • Exposure to enterprise data governance, MLOps, or LLMOps is preferred.
  • Databricks Asset Bundles and/or Terraform experience is a plus.

Benefits

  • Support, coaching, and feedback from engaging colleagues.
  • Opportunities to develop new skills and progress your career.
  • Freedom and flexibility to handle the role in a way that is right for you.
  • Individual progression planning, education, coaching, practical experience, and interdisciplinary knowledge exchange.

Tech Stack

Apache HadoopApache HiveApache SparkAWSAzureDatabricksGitPythonSQLTerraform

Categories

AI ApplicationsData Engineering
Ernst and Young

About Ernst and Young

10,000+ employees

Ernst & Young (EY) provides audit/assurance, tax, consulting, strategy and transactions services to enterprises, financial institutions, and public‑sector clients. Structured as a global network of partner‑owned member firms, it sells professional services on a fee basis, including a dedicated Financial Services Organization for banking, insurance, and capital markets. Headquartered in London, EY was formed in 1989 from the merger of Ernst & Whinney and Arthur Young, and operates in 150+ countries.

Contact me