14 days ago
Gurgaon, IndiaSenior
Responsibilities
- Design, build, operate, and upgrade batch and scheduled data pipelines using Airflow, Dagster, or Prefect.
- Own web and API ingestion, large-scale scraping, parsing, normalization, and file-merging systems.
- Build distributed processing architectures and analytical query layers across cloud data platforms.
- Improve entity matching, caching services, data accuracy, latency, and operational cost.
- Develop Python data access layers and APIs for downstream machine learning and product teams.
- Lead data quality initiatives including validation, retention policies, backfills, monitoring, and OpenTelemetry observability.
- Partner with data science, forecasting, and platform teams to safely ship features behind feature flags.
- Participate in code reviews, technical design documentation, mentoring, on-call rotations, incident response, and root-cause analysis.
Requirements
- 6+ years of software or data engineering experience, including 3+ years building and maintaining production data pipelines.
- Strong Python development skills and experience writing clean, tested code with pytest, nox, or tox.
- Deep workflow orchestration experience with Airflow or alternatives such as Dagster and Prefect.
- Hands-on experience with Spark and cloud data ecosystems including AWS EMR, Athena, S3, Databricks, Snowflake, and GCP.
- Solid SQL and data-modeling skills with practical experience using NoSQL databases, caching layers, and message queues.
- Experience with MongoDB, Redis, Kafka, or RabbitMQ.
- Experience deploying and managing containerized workloads on Kubernetes using Docker and Helm.
- Track record of owning production pipelines, including on-call participation, incident response, and root-cause analysis.
- Clear written technical communication and experience documenting architectural decisions.
- Preferred experience with large-scale web data collection, distributed crawler infrastructure, record linkage, entity resolution, fuzzy matching, NLP preprocessing, multi-cloud pipelines, and data pipelines feeding machine learning or forecasting systems.
- Preferred familiarity with spaCy and NLTK.
Tech Stack
Amazon DynamoDBApache AirflowApache KafkaApache SparkAWSDatabricksDockerGoogle BigQueryGoogle Cloud PlatformHelmKubernetesMongoDBNLTKpytestPythonRabbitMQRedisSeleniumSnowflakespaCySQL
Categories
Data Engineering
About Anaplan
Anaplan builds a cloud platform for enterprise planning, forecasting, and scenario modeling used by finance, supply chain, sales, and workforce teams. It sells subscription software and related services to large organizations that need connected, cross-functional planning. Founded in 2006 and headquartered in Miami, Florida, Anaplan was acquired by Thoma Bravo in 2022 after previously trading on the NYSE.