Principal Data Engineer (Apache Spark, dbt, Airflow)
ASX Limited2 months ago
Sydney, AustraliaStaff+
Responsibilities
- Evolve end-to-end data platform designs across ingestion, storage, transformation, and serving layers.
- Lead scalable, fault-tolerant, and cost-efficient data lake and data mesh architectures.
- Build reusable frameworks, libraries, and internal developer tooling for data pipeline development.
- Engage architecture, product, delivery, engineering, and business stakeholders to optimize data solutions.
- Ensure data pipelines meet engineering standards, process controls, compliance requirements, and reporting needs.
- Drive DataOps practices including CI/CD for data pipelines, infrastructure-as-code, automated testing, and observability.
- Design data solutions for the operational reliability requirements of critical infrastructure.
- Embed data privacy, lineage tracking, and regulatory controls into platform designs.
- Support embedded platform data engineers and help manage their deliveries and stakeholders.
Requirements
- 10+ years of data engineering experience.
- Proven experience designing and operating large-scale data platforms and pipelines in AWS.
- Experience with Apache Spark, dbt, and Airflow, including providing guidance on best practices.
- Deep expertise in distributed systems, batch and streaming data processing, and modern data architecture patterns.
- Experience with AWS cloud engineering and data platform concepts including data lakes, data warehouses, and data lakehouses.
- Experience with Redshift or Apache Iceberg, Terraform, and complex or semi-structured data formats including JSON, Parquet, Avro, and XML.
- Experience with design governance and end-to-end data platform design across large programs of work.
- At least 3 years in a lead data engineering role is preferred.
- Experience implementing data management capabilities in data platform frameworks is preferred.
- Familiarity with Confluence, JIRA, or ServiceNow is preferred.
- Strong ownership, customer focus, integrity, proactive leadership, and commitment to operational excellence are expected.
Benefits
- Flexible and hybrid working options are available, including consideration of part-time or other flexible arrangements.
- Successful candidates undergo background checks, including reference and police checks, during onboarding.
- Candidates must be legally authorized to work permanently in Australia without restrictions.
Tech Stack
Categories
Data Engineering
About ASX Limited
ASX Limited operates Australia’s primary securities exchange, providing listing, trading, clearing, settlement, market data, and connectivity across equities, derivatives, ETFs, and fixed income. It earns fees from listings, transactions, post‑trade services, and data/technology services for brokers, issuers, and investors. Founded in 1987 and headquartered in Sydney, ASX runs central counterparty clearing and depository infrastructure and is widely ranked among the world’s larger exchanges by market capitalization.