1 day ago
Sydney, AustraliaMid Level
Responsibilities
- Manage and monitor distributed systems, storage infrastructure, data processing platforms, and in-house data pipelines.
- Operate and troubleshoot large-scale data infrastructure safely and reliably.
- Drive systems automation and deployment processes for hardware and software solutions.
- Collaborate with systems and network engineers, traders, and developers to support and troubleshoot queries.
- Evaluate and implement innovative infrastructure solutions while keeping current with technology trends.
Requirements
- At least 2 years of experience managing large-scale, multi-petabyte data infrastructure in a similar role.
- Knowledge of Linux system administration and internals, with demonstrated troubleshooting ability in Linux environments.
- Experience with at least one of Kafka, HDFS, or Kubernetes.
- Working knowledge of Docker, Kubernetes, and Helm.
- Experience with data access technologies such as Dremio and Presto.
- Familiarity with workflow orchestration tools such as Airflow and Prefect.
- Exposure to AWS, GCP, or Azure cloud platforms.
Tech Stack
Categories
Data EngineeringSite Reliability