Principal Data Engineer (AWS, Databricks, Ai, ML Flow, Data Architecture, Apache Airflow)
Mastercard2 months ago
Pune, IndiaStaff+
Responsibilities
- Lead the architecture, design, and engineering of scalable batch, streaming, and event-driven data platforms.
- Build reliable data ingestion, transformation, enrichment, processing, and consumption pipelines.
- Develop governed lakehouse and modern data platform architectures using cloud-native technologies and distributed data processing frameworks.
- Build platforms supporting analytics, machine learning, generative AI, and agentic AI workloads.
- Create reusable data products, services, APIs, and self-service capabilities for experimentation and production delivery.
- Enable AI and machine learning lifecycle workflows including feature engineering, training, evaluation, deployment, and inference.
- Optimize data workloads for performance, scalability, reliability, resiliency, and cost efficiency.
- Define standards for data modeling, software development, testing, CI/CD, observability, and operational excellence.
- Implement data governance for quality, lineage, metadata, cataloging, access control, retention, compliance, security, and privacy.
- Evaluate emerging data and AI technologies through prototypes, proofs of concept, and technical assessments.
- Translate ambiguous research, innovation, and product requirements into practical architectures and incremental delivery plans.
- Provide hands-on technical leadership through architecture reviews, code reviews, troubleshooting, and mentoring.
- Partner with cross-functional teams to mature R&D initiatives into enterprise-grade production solutions.
- Communicate technical decisions, trade-offs, risks, dependencies, and roadmap recommendations to technical and business stakeholders.
- Drive adoption of modern data engineering practices, platform automation, and platform reliability disciplines.
Requirements
- Typically 12-18 years of relevant career experience, including significant ownership of complex enterprise data engineering solutions.
- Extensive experience designing, building, and operating large-scale production data platforms.
- Expert programming skills in Python and PySpark, with modern software engineering experience.
- Advanced SQL expertise covering data modeling, performance tuning, query optimization, and large-scale analytical processing.
- Strong hands-on experience with distributed data processing technologies and modern lakehouse platforms.
- Experience building and operating cloud-native data platforms in AWS, Azure, or other enterprise cloud environments.
- Knowledge of data lakes, lakehouses, data meshes, data products, and event-driven architecture patterns.
- Experience with cloud storage, data integration, streaming, serverless, observability, and security services.
- Experience building scalable batch and real-time data pipelines.
- Experience with workflow orchestration platforms such as Apache Airflow, Databricks Workflows, AWS Step Functions, or Azure Data Factory.
- Experience implementing CI/CD, automated testing, version control, Infrastructure as Code, and platform automation.
- Strong understanding of data governance, metadata management, lineage, quality frameworks, privacy controls, and access management.
- Experience resolving complex performance, reliability, scalability, and operational challenges.
- Ability to balance delivery speed, innovation, maintainability, security, and cost in architectural decisions.
- Ability to work effectively in R&D and innovation-focused environments with evolving priorities.
- Strong communication, collaboration, technical leadership, and mentoring skills.
- Bachelor's degree in Computer Science, Engineering, Information Systems, or a related technical discipline, or equivalent practical experience.
- Preferred experience supporting machine learning, generative AI, or agentic AI systems.
- Preferred experience with MLflow or comparable ML lifecycle tooling, Azure Machine Learning, Azure AI Foundry, Microsoft Fabric, Unity Catalog, Microsoft Purview, Kafka, Docker, Kubernetes, and Terraform.
- Preferred experience integrating structured, semi-structured, unstructured, and streaming data, and knowledge of responsible AI, model evaluation, AI platform observability, or regulated industries.
Tech Stack
Categories
Data Engineering
About Mastercard
Mastercard builds and operates a global payments network used by banks, merchants, fintechs, and governments, offering card processing, real-time payments, tokenization, and fraud/risk services. It generates revenue from transaction processing and assessment/service fees across more than 200 countries and territories. Founded in 1966 and headquartered in Purchase, New York, Mastercard is a public company listed on the NYSE (ticker: MA).