
Senior Developer, Data Engineer
Intercontinental Exchange1 day ago
Hyderābād, IndiaSenior
Responsibilities
- Own Apache Airflow DAGs and orchestration workflows across ingestion, transformation, validation, and delivery.
- Operate Airflow on Kubernetes using KubernetesExecutor or CeleryKubernetesExecutor, including Helm upgrades, pod templates, resource tuning, and scheduler performance management.
- Build and maintain production-scale batch, incremental, ETL, and ELT pipelines from APIs, message queues, databases, and object storage.
- Implement schema validation, row-count reconciliation, anomaly detection, freshness monitoring, and other data-quality controls.
- Design and maintain Apache Iceberg lakehouse architecture using Bronze, Silver, and Gold medallion layers.
- Use Databricks for cluster compute, Delta Lake workflows, Unity Catalog, and scheduled jobs.
- Build documented dbt projects with models, sources, tests, snapshots, seeds, macros, and freshness checks.
- Develop reusable pipeline templates, custom Airflow operators, and utility libraries for the shared platform.
- Partner with data scientists and ML engineers on feature pipelines and reproducible training datasets.
- Instrument pipelines with metrics, dashboards, and alerts; maintain runbooks and participate in on-call coverage.
- Contribute to architecture discussions, tooling evaluations, engineering standards, and team mentoring.
Requirements
- At least three years of experience running Apache Airflow in production.
- Expert knowledge of Airflow scheduler internals, executor types, XComs, Connections, Variables, pools, and the TaskFlow API.
- Production experience deploying and operating Airflow on Kubernetes with KubernetesExecutor or CeleryKubernetesExecutor.
- Strong Python skills for building tested DAGs, custom operators and sensors, cross-DAG triggers, and parameterized pipelines.
- Demonstrated experience building batch and incremental ETL or ELT pipelines at scale with data-quality validation and lineage capture.
- Practical lakehouse architecture experience, including medallion models and reliable downstream data availability.
- Hands-on Apache Iceberg experience, including snapshot isolation, partition evolution, hidden partitioning, time-travel queries, and catalog environments.
- Practical Databricks experience with cluster management, Delta Lake, Workflows or Jobs, and Unity Catalog.
- Confident building dbt projects with models, sources, tests, snapshots, seeds, macros, documentation, Airflow integration, and freshness checks.
- Strong engineering-grade Python experience with pandas, PySpark, SQLAlchemy, and Pydantic.
- Preferred experience with Kafka, Kafka Connect, Flink, Apache Spark, Kubernetes, OpenShift, Helm, RBAC, and namespace management.
- Familiarity with data contracts, DataHub or Collibra, Great Expectations or Soda, and managed data services on AWS, Azure, or GCP.
- Experience with S3, Glue, ADLS, Synapse, GCS, Dataproc, Terraform, or Pulumi is preferred.
- Understanding of feature stores and pipelines serving model training and inference is preferred.
- Experience with Databricks Asset Bundles or MLflow is a welcome bonus.
Tech Stack
Apache AirflowApache FlinkApache KafkaApache SparkAWSAzureDatabricksdbtGoogle Cloud PlatformHelmKubernetesMLflowOpenShiftPandasPythonTerraform
Categories
Data Engineering
About Intercontinental Exchange
Intercontinental Exchange operates global exchanges and clearing houses, offers fixed income and commodities trading platforms, and sells market data, listings, and mortgage technology to financial institutions and lenders. Its revenue comes from transaction and clearing fees, data subscriptions, listings, and software. Founded in 2000 and headquartered in Atlanta, it owns the New York Stock Exchange and ICE Mortgage Technology, and is publicly traded on the NYSE under the ticker ICE.