AVP - System Development Manager - Data Platform - LME
Hong Kong Exchanges and Clearing Limited1 day ago
Shenzhen, ChinaStaff+
Responsibilities
- Own the design, build, operation, and improvement of major data-platform components across streaming, batch, storage, serving, and orchestration.
- Operate Kafka clusters, schema registries, Kafka Connect connectors, and streaming infrastructure, including tuning, monitoring, rebalancing, and disaster recovery.
- Build and optimize Spark batch jobs and Flink or Spark Structured Streaming pipelines, including reusable frameworks and tuning guides.
- Manage Iceberg tables and operate MinIO at scale, including maintenance, lifecycle rules, tiering, and performance tuning.
- Deploy and operate Trino, StarRocks, and ClickHouse clusters, including connector configuration, resource management, sharding, replication, materialized views, and query monitoring.
- Build and maintain Airflow or Dagster DAGs, custom operators, and sensors for platform operations.
- Implement platform-wide monitoring, dashboards, and alerting with OpenTelemetry, Prometheus, Grafana, and Loki.
- Manage Kubernetes operations, Helm charts, operator lifecycles, resource quotas, node affinity, and pod disruption budgets.
- Own ArgoCD application sets, Helm-based deployments, and development-to-production promotion pipelines using GitOps practices.
- Participate in on-call rotations and create post-mortems and runbooks.
- Mentor mid-level engineers through pairing, design discussions, and code reviews.
Requirements
- 6+ years of experience in data engineering, platform engineering, or backend infrastructure.
- Strong Kubernetes experience including Helm chart authoring, RBAC, network policies, persistent volumes and claims, and operators.
- Solid Kafka experience with topic design, consumer groups, offset management, lag monitoring, Kafka Connect, and schema registries such as Apicurio or Confluent.
- Solid Spark experience with the DataFrame or Dataset API, Spark SQL, performance tuning, and production troubleshooting.
- Working knowledge of Flink or Spark Structured Streaming for real-time pipelines.
- Practical Iceberg experience including table maintenance, time travel, and catalog integration.
- Hands-on Trino or Presto experience with connector configuration, query tuning, and resource-group management.
- Experience with an OLAP engine such as StarRocks, ClickHouse, or Doris, including table design, ingestion pipelines, and query optimization.
- Proficiency in Python and either Scala or Java.
- Solid CI/CD and GitOps experience with ArgoCD or Flux, Helm, and Docker.
- Experience with Airflow or Dagster for pipeline orchestration.
- OpenShift experience with SCC, Routes, ImageStreams, and BuildConfigs is preferred.
- dbt project experience for data transformation and modeling is preferred.
- Experience with Great Expectations, Soda, or Deequ is preferred.
- Kafka Streams or ksqlDB experience is preferred.
- Exposure to DataHub or Atlas for data discovery and lineage is preferred.
Benefits
- Permanent position based in Shenzhen, China, with a standard 40-hour workweek.
- Participation in an on-call rotation is required.
Tech Stack
Apache AirflowApache FlinkApache KafkaApache SparkClickHousedbtDockerGrafanaHelmJavaKubernetesOpenShiftPrestoPrometheusPythonScala
Categories
Data EngineeringDevOps