22 days ago
Remote, Malaysia or Kuala Lumpur, MalaysiaSenior
Responsibilities
- Design, build, test, deploy, and operate scalable batch and applicable streaming data pipelines.
- Develop Python, PySpark, and SQL solutions for ingesting, transforming, and processing large-scale datasets.
- Design analytical data models and reusable, trusted datasets and data products.
- Engineer and optimise AWS-based data solutions using services such as Amazon S3, AWS Glue, Amazon Athena, Amazon Redshift, and Amazon EMR.
- Implement data lake and lakehouse practices, including storage formats, partitioning, schema evolution, and query optimisation.
- Build robust ETL/ELT patterns with incremental processing, dependency management, idempotency, retries, backfills, and failure recovery.
- Implement workflow orchestration and scheduling using Airflow/MWAA, AWS Step Functions, or AWS Glue Workflows.
- Build automated data quality controls, monitoring, observability, and pipeline health controls.
- Optimise Spark workloads and distributed processing for performance, scalability, and cost.
- Implement data governance, metadata, lineage, cataloguing, security, and access controls.
- Collaborate with clients, business users, data scientists, and technical teams to translate requirements into data solutions and integrations.
- Investigate production data and pipeline issues, perform root-cause analysis, and improve reliability and operational efficiency.
- Contribute to code reviews, CI/CD, automated deployments, and compliance with architecture, testing, security, and quality standards.
Requirements
- Degree or advanced diploma in Computer Science, Data Engineering, Software Engineering, or an equivalent qualification.
- At least 6 years of experience in data engineering or developing data-intensive systems, including production data pipeline design, build, and support.
- Strong hands-on Python and PySpark experience for large-scale distributed data processing.
- Strong SQL skills, including complex transformations, joins, window functions, query optimisation, and performance tuning.
- Experience designing and operating scalable ETL/ELT pipelines, data models, data lakes/lakehouses, workflow orchestration, automated data quality, and data observability.
- Hands-on experience with AWS data services and understanding of Spark performance engineering, security, governance, metadata, lineage, and schema management.
- Experience with Git workflows, code reviews, CI/CD, automated deployment, and Python/PySpark unit, integration, and data-pipeline testing.
- Strong debugging, troubleshooting, root-cause analysis, communication, stakeholder management, analytical thinking, and problem-solving skills.
- Familiarity with infrastructure as code, cloud cost optimisation, Jira, Bitbucket, Jenkins, Checkmarx, Agile, Scrum, SDLC, and AI coding tools is advantageous.
Benefits
- Hybrid work arrangement with home and office working.
- Inclusive paid leave.
- People-first and inclusive workplace.
- Paid internship is listed in the posting.
- The role supports clients and teams across the APAC region.
Categories
Data Engineering
About Experian
Experian builds credit reporting, identity and fraud prevention, and data analytics software that help lenders, insurers, healthcare providers, and marketers make decisions; it also offers consumer credit monitoring and scores. The company earns via data licensing, decisioning software, and subscription services, operates in 30+ countries, is headquartered in Costa Mesa, California, and is a FTSE 100 public company listed on the London Stock Exchange (EXPN); it is one of the three major U.S. credit bureaus.