
EY - GDS Consulting - AI And DATA - Databricks Architect - Senior Manager
Ernst and Young18 days ago
Kochi, IndiaStaff+
Responsibilities
- Architect and deliver scalable Data and AI solutions on the Databricks Lakehouse platform across ingestion, transformation, storage, governance, and consumption layers.
- Design and optimize distributed data processing pipelines using Hadoop, Apache Spark, and PySpark.
- Build reusable Python frameworks and production-grade batch and streaming data pipelines.
- Define cloud data architecture on AWS and/or Azure, including storage, IAM, networking, security, monitoring, and Databricks integration.
- Own ETL/ELT design, data ingestion, transformation, orchestration, and data quality frameworks.
- Design dimensional data models using star and snowflake schemas, fact and dimension tables, and slowly changing dimensions.
- Build Agentic AI solutions with LLM integration, tool/function calling, agent workflows, guardrails, and enterprise data integration.
- Architect and implement Databricks-based RAG solutions covering document ingestion, chunking, embeddings, vector search, retrieval, grounding, and evaluation.
- Establish Git-based version control, code reviews, and CI/CD practices across development, test, and production environments.
- Lead performance optimization and production deployment of Databricks jobs, workflows, and pipelines.
- Participate in RFP/RFI responses, technical proposals, solution shaping, effort estimation, and architecture presentations.
- Lead customer workshops, translate business requirements into technical architectures, and mentor engineering teams.
Requirements
- 13+ years of experience in Data Engineering or Data Architecture.
- Strong hands-on expertise with Databricks Lakehouse, Delta Lake, Databricks SQL, Unity Catalog, Jobs/Workflows, performance optimization, and production deployments.
- Strong knowledge of the Hadoop ecosystem, HDFS, Hive, Apache Spark, and PySpark.
- Strong Python proficiency for scalable data pipelines and frameworks.
- Experience with AWS and/or Azure cloud architecture, including storage, IAM, networking, security, monitoring, and cloud-native integrations.
- Expertise in ETL/ELT, batch and streaming processing, ingestion, transformation, orchestration, data quality, and scalable pipeline architecture.
- Strong SQL and dimensional modeling skills, including star/snowflake schemas, fact/dimension models, SCD, and analytical data models.
- Hands-on experience building Agentic AI and Databricks RAG solutions, including LLMs, tool/function calling, agent workflows, guardrails, embeddings, vector search, retrieval, grounding, and evaluation.
- Knowledge of Git, branching strategies, pull requests, code reviews, CI/CD pipelines, and multi-environment deployment.
- Ability to contribute to strategic RFP/RFI responses, solution shaping, technical proposals, effort estimates, architecture presentations, and customer workshops.
- Strong client-facing communication skills and ability to translate business requirements into scalable technical solutions.
- Consulting, systems integrator, or client-facing delivery experience is preferred.
- Relevant Databricks or cloud certifications, such as Databricks Data Engineer, Databricks Solutions Architect, AWS, or Azure certifications, are preferred.
- Exposure to enterprise data governance, MLOps, or LLMOps is preferred.
- Databricks Asset Bundles and/or Terraform experience is a plus.
Benefits
- Support, coaching, and feedback from engaging colleagues.
- Opportunities to develop new skills and progress your career.
- Freedom and flexibility to handle the role in a way that is right for you.
- Individual progression planning, education, coaching, practical experience, and interdisciplinary knowledge exchange.
Categories
AI ApplicationsData Engineering
About Ernst and Young
Ernst & Young (EY) provides audit/assurance, tax, consulting, strategy and transactions services to enterprises, financial institutions, and public‑sector clients. Structured as a global network of partner‑owned member firms, it sells professional services on a fee basis, including a dedicated Financial Services Organization for banking, insurance, and capital markets. Headquartered in London, EY was formed in 1989 from the merger of Ernst & Whinney and Arthur Young, and operates in 150+ countries.