Principal Data Engineer (Apache Spark, dbt, Airflow)
ASX Limited2 hours ago
Sydney, AustraliaStaff+
Responsibilities
- Evolve end-to-end data platform designs across ingestion, storage, transformation, and serving layers.
- Lead scalable, fault-tolerant, and cost-efficient data lake and data mesh architectures.
- Build reusable frameworks, libraries, and internal developer tooling for data pipeline development.
- Engage architecture, product, delivery, engineering, and business stakeholders to optimize data solutions.
- Ensure data pipelines meet engineering standards, process controls, compliance requirements, and reporting needs.
- Drive DataOps practices including CI/CD for data pipelines, infrastructure-as-code, automated testing, and observability.
- Design data solutions for the operational reliability requirements of critical infrastructure.
- Embed data privacy, lineage tracking, and regulatory controls into platform designs.
- Support embedded platform data engineers and help manage their deliveries and stakeholders.
Requirements
- 10+ years of data engineering experience.
- Proven experience designing and operating large-scale data platforms and pipelines in AWS.
- Experience with Apache Spark, dbt, and Airflow, including providing guidance on best practices.
- Deep expertise in distributed systems, batch and streaming data processing, and modern data architecture patterns.
- Experience with AWS cloud engineering and data platform concepts including data lakes, data warehouses, and data lakehouses.
- Experience with Redshift or Apache Iceberg, Terraform, and complex or semi-structured data formats including JSON, Parquet, Avro, and XML.
- Experience with design governance and end-to-end data platform design across large programs of work.
- At least 3 years in a lead data engineering role is preferred.
- Experience implementing data management capabilities in data platform frameworks is preferred.
- Familiarity with Confluence, JIRA, or ServiceNow is preferred.
- Strong ownership, customer focus, integrity, proactive leadership, and commitment to operational excellence are expected.
Benefits
- Flexible and hybrid working options are available, including consideration of part-time or other flexible arrangements.
- Successful candidates undergo background checks, including reference and police checks, during onboarding.
- Candidates must be legally authorized to work permanently in Australia without restrictions.
Tech Stack
Categories
Data Engineering