Senior Data Engineer - Product
CoStar Group3 months ago
Sydney, AustraliaSenior
Responsibilities
- Design and build scalable, reusable data solutions and assets on a modern Lakehouse platform.
- Develop and maintain batch and streaming data pipelines using DBT, Apache Airflow, Snowflake, Databricks, and AWS services.
- Build real-time, event-driven pipelines with Apache Kafka and Apache Flink.
- Integrate internal and external data sources through SaaS solutions, APIs, direct connections, and streaming methods.
- Architect cloud-native data solutions on AWS, Snowflake, and Databricks with attention to cost efficiency.
- Build and optimize Apache Iceberg Lakehouse tables, including schema evolution, partitioning, and performance.
- Apply modern data modeling practices and work with semi-structured and unstructured data.
- Optimize complex SQL and Spark workloads for performance and cost.
- Support data quality and governance initiatives.
- Collaborate with data scientists, analysts, product managers, and engineers to deliver business-value-driven data solutions.
- Identify opportunities to improve data processes, reduce costs, and increase reliability.
Requirements
- At least 5+ years of hands-on commercial experience in software or data engineering.
- At least 4+ years of strong development experience in SQL and Python.
- Experience with data integration and orchestration tools including Apache Airflow, DBT, and AWS Glue.
- Expertise in modern cloud data platforms and architectures such as Snowflake, Databricks, and Lakehouse architecture.
- Experience designing and building data solutions on AWS cloud infrastructure.
- Strong data modeling skills and experience evolving models in an agile environment.
- Hands-on experience building streaming and real-time pipelines with Apache Kafka and Apache Flink.
- Knowledge of infrastructure-as-code tools such as CDK, CloudFormation, or Terraform.
- Degree in computer science, engineering, mathematics, statistics, or a related technical field.
Benefits
- Generous compensation and performance-based incentives.
- Outstanding working environment and collaborative culture.
- Extra days of annual leave.
- Leadership development programs.
- LinkedIn Learning.
- Perkbox discounts.
- Inclusive and equitable recruitment process with workplace adjustments available for candidates with disabilities.
Tech Stack
Categories
Data Engineering
About CoStar Group
CoStar Group builds subscription databases, analytics, and online marketplaces for commercial and residential real estate professionals, investors, and advertisers. Its portfolio includes CoStar (research and data), LoopNet (commercial listings), and Apartments.com (rental listings), generating revenue from subscriptions, listings, and advertising. The company is publicly traded on NASDAQ (CSGP) and is headquartered in Arlington, Virginia, with inclusion in the S&P 500 index.