Principal Data Engineer (AWS, Databricks, Ai, ML Flow, Data Architecture, Apache Airflow)
Mastercard2 months ago
Pune, IndiaStaff+
Responsibilities
- Lead the architecture, design, and engineering of scalable batch, streaming, and event-driven data platforms.
- Build reliable data ingestion, transformation, enrichment, processing, and consumption pipelines.
- Develop governed lakehouse and modern data platform architectures using cloud-native and distributed data processing technologies.
- Operate platforms supporting analytics, machine learning, generative AI, and agentic AI workloads.
- Create reusable data products, services, APIs, and self-service capabilities.
- Enable feature engineering, training, evaluation, deployment, and inference pipelines across the AI/ML lifecycle.
- Optimize workloads for performance, scalability, reliability, resiliency, and cost efficiency.
- Define standards for data modeling, software development, testing, CI/CD, observability, and operational excellence.
- Implement data quality, lineage, metadata, cataloging, access control, retention, compliance, security, and privacy capabilities.
- Evaluate emerging data and AI technologies through prototypes, proofs of concept, and technical assessments.
- Translate research, innovation, and product requirements into practical architectures and delivery plans.
- Provide hands-on technical leadership through architecture reviews, code reviews, troubleshooting, mentoring, and stakeholder communication.
- Partner with cross-functional teams to mature R&D initiatives into enterprise-grade production solutions.
- Drive adoption of modern data engineering practices, platform automation, and reliability disciplines.
Requirements
- Typically 12–18 years of overall relevant career experience, including significant ownership of complex enterprise data engineering solutions.
- Extensive experience designing, building, and operating large-scale production data platforms.
- Expert programming skills in Python and PySpark, with advanced SQL expertise.
- Strong experience with Apache Spark, lakehouse platforms, cloud-native data platforms, scalable batch and real-time pipelines, and distributed data processing.
- Experience with cloud storage, data integration, streaming, serverless, observability, and security services in AWS, Azure, or other enterprise cloud environments.
- Experience with Apache Airflow, Databricks Workflows, AWS Step Functions, Azure Data Factory, or comparable workflow orchestration platforms.
- Experience with automated testing, version control, Infrastructure as Code, CI/CD, and platform automation.
- Strong understanding of data architecture, governance, metadata management, lineage, data quality, privacy controls, and access management.
- Ability to diagnose complex performance, reliability, scalability, and operational issues and make balanced architectural decisions.
- Ability to work effectively in R&D and innovation-focused environments with evolving priorities.
- Strong communication, collaboration, technical leadership, and mentoring skills.
- Bachelor’s degree in Computer Science, Engineering, Information Systems, or a related technical discipline, or equivalent practical experience.
- Preferred experience includes supporting machine learning, generative AI, or agentic AI systems and using MLflow or comparable ML lifecycle tooling.
- Preferred experience includes Azure Machine Learning, Azure AI Foundry, Microsoft Fabric, Unity Catalog, Microsoft Purview, Kafka, Docker, Kubernetes, Terraform, structured and unstructured data integration, responsible AI, model evaluation, and AI platform observability.
- Experience in payments, financial services, or another regulated industry is preferred.
Tech Stack
Categories
Data Engineering
About Mastercard
Mastercard builds and operates a global payments network used by banks, merchants, fintechs, and governments, offering card processing, real-time payments, tokenization, and fraud/risk services. It generates revenue from transaction processing and assessment/service fees across more than 200 countries and territories. Founded in 1966 and headquartered in Purchase, New York, Mastercard is a public company listed on the NYSE (ticker: MA).