1 day ago
Chennai, IndiaSenior
Responsibilities
- Design, develop, and optimize scalable Python and SQL ELT/ETL pipelines for real-time, near-real-time, and batch processing.
- Architect and manage enterprise-scale Snowflake environments, including warehouses, databases, schemas, RBAC, resource monitors, clustering, and performance optimization.
- Implement Snowflake Tasks, Streams, Snowpipe, External Tables, incremental loads, CDC, SCD, schema evolution, and orchestration patterns.
- Build cloud-native data solutions using Azure storage, compute, integration, and governance services.
- Design data warehouses, data marts, semantic layers, and modern dimensional, Kimball, and Data Vault models.
- Develop AI and generative AI solutions and partner with BI, ML, Product, Data Science, and Business teams.
- Implement CI/CD, DevSecOps controls, infrastructure as code, secret scanning, dependency scanning, and policy-as-code.
- Implement data unit testing, schema validation, contract enforcement, profiling, quality dashboards, lineage, and SLA monitoring.
- Lead design and code reviews, contribute to data platform roadmap discussions, and mentor junior engineers.
Requirements
- 6–10 years of professional data engineering experience.
- Strong Python expertise, including pandas, asyncio, object-oriented programming, typing, packaging, and pytest.
- Hands-on enterprise-scale Snowflake experience and advanced SQL skills involving joins, window functions, CTEs, and performance tuning.
- Strong experience with AWS, Azure, or GCP and deep knowledge of data warehousing, dimensional modeling, star schemas, data marts, CDC, and SCD.
- Experience with GitHub Actions, Azure DevOps, or GitLab CI; Terraform or Bicep; and DevSecOps practices such as secrets management, IAM, vulnerability scanning, and zero-trust principles.
- Experience with orchestration tools such as Airflow, ADF, Prefect, or Step Functions.
- Experience with dbt, streaming platforms such as Kafka, Kinesis, or Pub/Sub, and lineage tools such as OpenLineage or Marquez.
- Familiarity with Lakehouse architectures including Delta Lake, Iceberg, or Hudi, plus MLOps concepts and feature stores.
- Knowledge of cost governance and FinOps best practices, along with prior leadership or mentorship experience.
- Excellent communication and stakeholder management skills.
Benefits
- Flexible working environment.
- Volunteer time off, LinkedIn Learning, and an Employee Assistance Program.
Tech Stack
Apache AirflowApache KafkaAWSAzuredbtGitHub ActionsGitLab CI/CDGoogle Cloud PlatformPandaspytestPythonSnowflakeSQLTerraform
Categories
Data Engineering