3 days ago
Bengaluru, IndiaStaff+
Responsibilities
- Define and enforce architectural standards for real-time Spark Structured Streaming pipelines.
- Lead architecture for new application onboarding, including data flows, collection models, and CDC subscriptions.
- Evaluate and improve pipeline architectures for performance, scalability, and maintainability.
- Define MongoDB collection schemas, indexing strategies, aggregation approaches, and write patterns.
- Design Kafka topics, partition strategies, consumer configurations, and CDC subscriptions.
- Collaborate with HOST/mainframe, GFT, CDC squad, and platform engineering teams.
- Create and maintain high-level designs, data flow documents, and Architecture Decision Records.
- Review development team designs and provide architectural guidance.
- Support production deployments across INT, PRE, and PRO environments on Amazon EMR.
Requirements
- Scala and Apache Spark expertise, including Structured Streaming internals, execution models, and performance tuning.
- Experience designing end-to-end real-time data pipelines and data architectures.
- MongoDB experience covering collection design, indexing, aggregation pipelines, and Atlas architecture.
- Kafka experience covering topic design, partition strategies, consumer groups, and Confluent Cloud.
- AWS experience with EMR, S3, and CloudWatch.
- At least 6 years of data engineering experience, including at least 2 years in a technical architecture or lead role.
- 15 years of full-time education.
- Preferred experience with IBM CDC, Change Data Capture, Delta Lake on S3, banking data, Jenkins, SonarQube, Bitbucket, Java, mainframe DB2 data structures, IBAN calculation standards, and MongoDB Atlas cross-cluster architectures.
Tech Stack
Categories
Data Engineering
About Accenture
Accenture is a global professional services firm providing management consulting, systems integration and technology, cybersecurity, and business process outsourcing for enterprises and governments. It operates a services-driven model delivering projects and managed services, often with major cloud and software partners, across industries worldwide. Headquartered in Dublin and publicly traded on the NYSE (ACN), it originated as Andersen Consulting and adopted the Accenture name in 2001.