Intercontinental Exchange

Senior Developer, Data Engineer

Intercontinental Exchange
Apply
1 day ago
Hyderābād, IndiaSenior

Responsibilities

  • Own Apache Airflow end to end, including complex DAGs, orchestration patterns, testing, reviews, alerting, and operational standards.
  • Deploy and operate Airflow on Kubernetes using KubernetesExecutor or CeleryKubernetesExecutor, including Helm upgrades, pod configuration, resource tuning, and scheduler performance.
  • Build and maintain scalable batch, incremental, ETL, and ELT pipelines across APIs, message queues, databases, and object storage.
  • Implement data-quality validation, schema checks, reconciliation, anomaly detection, freshness monitoring, and lineage capture within pipelines.
  • Design and maintain Apache Iceberg lakehouse architecture using Bronze, Silver, and Gold data layers, including schema evolution, partitioning, and time-travel queries.
  • Manage Databricks cluster compute, Delta Lake workflows, Unity Catalog, and scheduled jobs with attention to cost and usability.
  • Build and maintain dbt projects with models, sources, tests, snapshots, seeds, macros, documentation, and freshness checks integrated with Airflow.
  • Create reusable pipeline templates, custom Airflow operators, and utility libraries for the shared data platform.
  • Deliver feature pipelines and reproducible training datasets in partnership with data scientists and ML engineers.
  • Instrument pipelines with metrics, dashboards, and alerts; maintain runbooks and participate in on-call coverage.
  • Contribute to architecture and tooling evaluations, engineering standards, and team mentoring.

Requirements

  • At least three years of experience running Apache Airflow in production, including scheduler internals, executor types, XComs, connections, variables, pools, and the TaskFlow API.
  • Hands-on experience deploying and operating Airflow on Kubernetes with KubernetesExecutor or CeleryKubernetesExecutor, including pod templates, resource tuning, persistent volume claims, Helm management, and upgrades.
  • Strong Python experience for DAG authoring, custom operators and sensors, dynamic task mapping, cross-DAG triggers, parameterized pipelines, testing, and packaging.
  • Demonstrated experience building and operating batch and incremental ETL or ELT pipelines at scale with data-quality validation and lineage capture.
  • Practical lakehouse architecture experience, including medallion models and reliable downstream data availability.
  • Hands-on Apache Iceberg experience with snapshot isolation, partition evolution, hidden partitioning, and time-travel queries in a real catalog environment.
  • Practical Databricks experience with cluster management, Delta Lake, Workflows or Jobs, and Unity Catalog.
  • Confident building dbt projects with models, sources, tests, snapshots, seeds, macros, documentation, and Airflow integration.
  • Comfort with pandas, PySpark, SQLAlchemy, and Pydantic.
  • Preferred experience with Apache Kafka, Kafka Connect, Flink, Apache Spark, Kubernetes, OpenShift, Helm, RBAC, and namespace management.
  • Preferred familiarity with data contracts, DataHub or Collibra, Great Expectations or Soda, and managed data services on AWS, Azure, or GCP.
  • Preferred experience with S3, Glue, ADLS, Synapse, GCS, Dataproc, Terraform, Pulumi, feature stores, and ML feature engineering.
  • Databricks Asset Bundles or MLflow experience is a bonus.

Tech Stack

Apache AirflowApache FlinkApache KafkaApache SparkAWSAzureDatabricksdbtGoogle Cloud PlatformHelmKubernetesMLflowOpenShiftPandasPythonTerraform

Categories

Data Engineering
Intercontinental Exchange

About Intercontinental Exchange

10,000+ employees

Intercontinental Exchange operates global exchanges and clearing houses, offers fixed income and commodities trading platforms, and sells market data, listings, and mortgage technology to financial institutions and lenders. Its revenue comes from transaction and clearing fees, data subscriptions, listings, and software. Founded in 2000 and headquartered in Atlanta, it owns the New York Stock Exchange and ICE Mortgage Technology, and is publicly traded on the NYSE under the ticker ICE.

Contact me