The Trade Desk

Software Engineer II, DataProc

The Trade Desk
Apply
2 days ago
London, United KingdomMid Level

Responsibilities

  • Design, build, and operate data orchestration and workflow platform services and libraries using Airflow and TaskService.
  • Develop, scale, and optimize distributed data processing jobs running on Spark across Amazon EMR, Kubernetes, and Databricks.
  • Partner with engineering teams to migrate and modernize large-scale data workloads.
  • Improve platform reliability and observability through metrics, alerting, tracing, and on-call tooling.
  • Build and maintain internal tools and lightweight UIs for monitoring, debugging, and operating data pipelines.
  • Own projects end to end through design, code review, deployment, rollout, and production support.
  • Debug complex issues in large-scale distributed production systems and contribute to company-wide data pipeline practices.

Requirements

  • At least 2 years of software development or engineering experience.
  • Production experience with Python, Scala, or Java.
  • Experience building or operating distributed data processing systems such as Spark, Databricks, or Hadoop, and/or workflow orchestration tools such as Airflow.
  • Comfort working with AWS, Kubernetes, and CI/CD pipelines.
  • Knowledge of data structures, algorithms, and distributed systems design.
  • Strong debugging and troubleshooting skills in large-scale production systems.
  • A bachelor’s or master’s-level degree in computer science or a relevant engineering field, or equivalent experience.
  • Preferred experience includes SQL, data warehousing, Snowflake, Vertica, Kubernetes operators, cluster administration, OpenTelemetry, Grafana, Prometheus, production on-call responsibilities, and machine learning pipelines or ad-tech systems.

Tech Stack

Apache AirflowApache HadoopApache SparkAWSDatabricksGrafanaJavaKubernetesPrometheusPythonScalaSnowflakeSQL

Categories

BackendData Engineering
Contact me