Egen

Lead Data Engineer- GCP

Egen
Apply
4 hours ago
Hyderābād, IndiaStaff+

Responsibilities

  • Gather customer requirements and participate in discovery, solution design, and architecture activities.
  • Design, develop, test, and maintain scalable Python-based ETL and ELT data pipelines.
  • Architect enterprise data solutions using GCP services, Kafka, GKE, Apigee, dbt, and related technologies.
  • Implement data ingestion, transformation, cleansing, validation, monitoring, and data quality controls.
  • Use Dataflow, Cloud Functions, BigQuery, Cloud Composer, Google Cloud Storage, IAM, and Cloud Run for data engineering solutions.
  • Collaborate with data scientists, analysts, and engineering teams to deliver efficient data solutions.
  • Manage version control with GitHub and participate in data project deployment pipelines.
  • Write complex SQL queries for extraction and validation from SQL Server, Oracle, and PostgreSQL databases.
  • Document pipeline designs, data flow diagrams, and operational support procedures.
  • Provide post-delivery customer support.

Requirements

  • 10+ years of hands-on Python experience in backend or data engineering projects.
  • Strong working experience with Google Cloud Platform, especially Dataflow, BigQuery, Cloud Functions, and Cloud Composer.
  • Strong understanding of data pipeline architecture, data integration, and transformation techniques.
  • Experience with Apache Spark, PySpark, Kafka, Redis, FastAPI, Airflow, and GCP Composer DAGs.
  • Strong SQL experience with at least one enterprise database such as SQL Server, Oracle, or PostgreSQL.
  • Experience with GitHub and CI/CD practices.
  • Experience migrating data from on-premise sources to cloud platforms.
  • Bachelor's degree in Computer Science or a related field, or equivalent experience.
  • Experience with Snowflake, Databricks, Azure Data Factory, other Azure data engineering tools, or AWS services is optional.
  • Strong problem-solving, analytical, communication, and collaboration skills.

Tech Stack

Apache AirflowApache KafkaApache SparkAWSDatabricksdbtFastAPIGoogle BigQueryGoogle CloudGoogle Cloud PlatformMicrosoft SQL ServerMongoDBPostgreSQLPythonRedisSnowflakeSQL

Categories

Data Engineering
Egen

About Egen

1,001-5,000 employees

Egen is a privately held IT services firm that designs and builds cloud, data, analytics, AI, and platform solutions for enterprises and public-sector clients. It delivers consulting, engineering, and managed services across major ecosystems, including Google Cloud and Salesforce. Founded in 2000 and headquartered in Naperville, Illinois, Egen operates across North America, Europe, and India.

Contact me