Senior Data Engineer – Python, BigQuery and DBT
ISS Governance1 month ago
Gurgaon, IndiaSenior
Responsibilities
- Design and build dimensional, normalized, and denormalized data models with slowly changing dimensions.
- Develop and maintain modular, tested, documented dbt transformations on BigQuery using macros and incremental strategies.
- Design, build, and maintain reliable ELT pipelines for financial data and external vendor feeds.
- Build ingestion frameworks that handle varied file formats, schema changes, validation, reconciliation, and error recovery.
- Orchestrate workflows with Apache Airflow and Cloud Composer, including dependencies, retries, SLAs, and alerting.
- Implement full-load, incremental-load, backfill, idempotency, and pipeline-recovery patterns.
- Create automated data reconciliation, validation, quality checks, monitoring, and freshness alerts.
- Optimize BigQuery partitioning, clustering, materialisation, caching, query performance, and storage costs.
- Build and support FastAPI REST APIs for internal and external data consumption.
- Support CI/CD pipelines, disaster recovery planning, testing, and documentation for data infrastructure.
- Maintain data lineage, documentation, cataloguing, governance, and auditable platform practices.
- Troubleshoot production data issues, perform root-cause analysis, and collaborate with operations, product, and engineering teams.
- Explore Generative AI, LLM-based tooling, RAG patterns, and agentic AI for engineering workflows and automation.
Requirements
- 9+ years of experience in data engineering, database development, or a related role.
- Bachelor’s or Master’s degree in Computer Science, Information Technology, or a related field.
- Experience in financial services or fintech, particularly securities master data, corporate actions, index calculations, or market data vendor feeds.
- Experience modernizing legacy data platforms and pipelines with modern ELT approaches.
- Strong data modelling and SQL skills, including dimensional modelling, star and snowflake schemas, slowly changing dimensions, window functions, CTEs, recursive queries, query-plan analysis, and performance tuning.
- Hands-on experience with dbt, including modelling, transformations, tests, documentation, macros, and incremental models.
- Experience with BigQuery or another cloud data warehouse such as Snowflake or Redshift, with willingness to work on BigQuery.
- Experience building Python data pipelines with Apache Airflow, Cloud Composer, or another workflow orchestration tool.
- Understanding of ELT/ETL patterns, data lake and warehouse architectures, lakehouse concepts, data reconciliation, governance, lineage, cataloguing, access control, and data quality.
- Knowledge of Parquet, Avro, JSON, and CSV formats and structured and semi-structured data processing.
- Familiarity with Git, CI/CD, REST APIs, FastAPI or Flask, Datadog, relational databases, SSIS or Informatica, CDC, and event-driven data patterns.
- Nice-to-have experience with Google Cloud Platform services beyond BigQuery, including Cloud Run, Cloud Composer, Cloud SQL, and Cloud Storage, plus API gateways such as Apigee.
- Understanding of exchange calendars, business-day logic, Generative AI concepts, large language models, retrieval-augmented generation, and agentic AI patterns.
- Strong ownership, problem-solving, communication, independent working ability, and comfort collaborating in globally distributed teams.
Benefits
- The role is based in Gurgaon and involves working with a globally distributed team across time zones.
- The company emphasizes diversity, inclusion, creativity, innovation, career growth, and meaningful impact.
- ISS STOXX operates across 20 countries with approximately 4,000 professionals and serves about 5,000 clients.
Tech Stack
Amazon RedshiftApache AirflowDatadogdbtFastAPIFlaskGitGoogle BigQueryGoogle Cloud PlatformInformaticaMicrosoft SQL ServerPostgreSQLPythonSnowflake
Categories
Data Engineering