7 days ago
Sydney, AustraliaSenior
Base Salary
$90k - $100k/yr
Responsibilities
- Design and build end-to-end Big Data and ETL pipelines using Apache Spark, Hive, PySpark, and Spark SQL.
- Develop transformation logic, populate SCD tables, and maintain historical and active records using SCD Type 2 logic.
- Perform data completeness, validation, schema validation, duplication, and accuracy checks for production data feeds.
- Build shell scripts and Control-M jobs for ingestion from edge nodes to raw, landing, and transformation layers.
- Deploy code across development, system integration testing, and production-related environments through Jenkins and Bitbucket pipelines.
- Analyze production issues, review Spark logs, and provide root-cause analysis after sprint releases.
- Participate in design and implementation walkthroughs, technical PIV activities, functional testing, and production deployment changes.
- Collaborate with clients and internal teams to assess change feasibility and deliver requested features.
- Use Jira, Confluence, and Bitbucket for project management and team collaboration.
- Plan, prioritize, schedule, track tasks, and independently manage delivery activities for clients.
Requirements
- At least 6 years of experience designing Big Data pipelines and building ETL solutions.
- Experience with Apache Spark, Sqoop, PySpark, Spark SQL, Hive, and Sterling.
- Experience with MySQL, Oracle, Mongo, MS SQL, and IBM-DB databases.
- Proficiency in Python and experience building shell scripts.
- Expertise setting up and managing continuous integration and continuous deployment pipelines with Bitbucket and Jenkins.
- Familiarity with Control-M for job scheduling and pipeline orchestration.
- Experience using monitoring tools and Spark logs for troubleshooting and root-cause analysis.
- Understanding of security best practices and compliance requirements.
- Banking, BFSI, cards, and payments domain knowledge.
- Experience with data quality validation, SCD Type 2 processing, production deployment, and client-facing delivery.
Tech Stack
Categories
Data Engineering
About Cognizant
Cognizant is a public IT services and consulting firm that designs, builds, and runs enterprise technology, including digital engineering, cloud modernization, data/AI, and managed services. It sells consulting, systems integration, and outsourcing on multi-year engagements to large enterprises in healthcare, banking, retail, communications, and manufacturing. Founded in 1994 and headquartered in Teaneck, New Jersey, Cognizant is NASDAQ-listed (CTSH) and a Fortune 500 company.