Airflow Platform Engineer
Jade Global2 months ago
Pune, India or Hyderābād, IndiaMid Level / Senior
Responsibilities
- Design, develop, optimize, troubleshoot, and maintain enterprise Apache Airflow DAGs using advanced orchestration patterns.
- Enable modern data workflows, including DBT pipelines, and provide reusable templates, best practices, and onboarding support.
- Operate and support Airflow deployed on AKS, including its scheduler, webserver/API, workers, and metadata services.
- Diagnose Kubernetes issues involving pods, networking, storage, and RBAC, as well as Airflow scheduling and resource problems.
- Support CI/CD pipelines for DAG and container deployments and improve platform stability, monitoring, and alerting.
- Develop and maintain unit, integration, and DAG validation tests, including operator mocking and dependency validation.
Requirements
- 4–6 years of experience.
- Strong production experience with Apache Airflow 3.x or later and Airflow DAG development.
- Working knowledge of Kubernetes, preferably AKS, for supporting the Airflow runtime.
- Expert-level Python skills for building, debugging, and maintaining production workflows and supporting libraries.
- Expertise writing unit, integration, and DAG validation tests for Airflow workflows in CI/CD environments.
- Experience with DBT, modern data pipelines, and CI/CD tools including Git, Jenkins, and Docker.
- Familiarity with Azure or other cloud platforms.
- Preferred: experience with Astronomer/Astro Private Cloud, Snowflake, APIs, Splunk, Prometheus, and Azure Key Vault.
Categories
Data EngineeringDevOps