
Software Engineer II, DataProc
The Trade Desk2 days ago
London, United KingdomMid Level
Responsibilities
- Design, build, and operate data orchestration and workflow platform services and libraries using Airflow and TaskService.
- Develop, scale, and optimize distributed data processing jobs running on Spark across Amazon EMR, Kubernetes, and Databricks.
- Partner with engineering teams to migrate and modernize large-scale data workloads.
- Improve platform reliability and observability through metrics, alerting, tracing, and on-call tooling.
- Build and maintain internal tools and lightweight UIs for monitoring, debugging, and operating data pipelines.
- Own projects end to end through design, code review, deployment, rollout, and production support.
- Debug complex issues in large-scale distributed production systems and contribute to company-wide data pipeline practices.
Requirements
- At least 2 years of software development or engineering experience.
- Production experience with Python, Scala, or Java.
- Experience building or operating distributed data processing systems such as Spark, Databricks, or Hadoop, and/or workflow orchestration tools such as Airflow.
- Comfort working with AWS, Kubernetes, and CI/CD pipelines.
- Knowledge of data structures, algorithms, and distributed systems design.
- Strong debugging and troubleshooting skills in large-scale production systems.
- A bachelor’s or master’s-level degree in computer science or a relevant engineering field, or equivalent experience.
- Preferred experience includes SQL, data warehousing, Snowflake, Vertica, Kubernetes operators, cluster administration, OpenTelemetry, Grafana, Prometheus, production on-call responsibilities, and machine learning pipelines or ad-tech systems.
Tech Stack
Apache AirflowApache HadoopApache SparkAWSDatabricksGrafanaJavaKubernetesPrometheusPythonScalaSnowflakeSQL
Categories
BackendData Engineering