Data Engineer with experience to Python, Airflow, Databricks & SQL

2 months ago

Chennai, IndiaMid Level

Responsibilities

Own end-to-end data flows from requirements and architecture through implementation and production operations, including data acquisition, dataset acceptance criteria, and Data Science integration.
Design and build scalable batch and real-time data pipelines and lakehouse solutions for large-scale data processing.
Explore and adopt technologies and patterns to scale and evolve the data ecosystem.
Partner with Data Science to productionize ML/AI workloads and integrate them into products.
Collaborate with cloud, DevOps, application, and client teams to deliver robust, secure, and scalable solutions.
Design, build, and own production-grade pipelines, including reliability, backfills, and SLA-driven delivery.

3+ years of backend engineering experience in production environments involving data-intensive systems, distributed processing, and databases.
B.Sc. or M.Sc. in Computer Science, Computer Engineering, or equivalent.
Proficiency in Python and working proficiency in Scala.
Strong experience with Spark, Databricks, and Kafka.
Experience with Airflow and building maintainable, operationalized DAGs.
Strong expertise with at least one major cloud provider, including Azure.
Strong SQL skills and experience with PostgreSQL plus a data lake/lakehouse, columnar database, or NoSQL system.
Experience with Delta Lake and/or Apache Iceberg, MLflow, Pandas or Polars, and data services/APIs such as FastAPI.
Proven experience designing, building, and owning production-grade batch and/or streaming data pipelines.
Strong English communication skills and the ability to work in a dynamic environment.
Preferred experience with containers, Kubernetes, Helm, CI/CD, LLM-powered agents, RAG, tool/function calling, LangChain, LangGraph, infrastructure as code, cloud security fundamentals, observability, distributed-workload optimization, React, and Node.js.