7 hours ago
Chennai, IndiaStaff+
Responsibilities
- Design and implement scalable, reliable, and secure data pipelines for structured and unstructured data.
- Build batch and near-real-time data ingestion frameworks and optimize data models for analytics, reporting, and machine learning workloads.
- Architect and maintain cloud-native data platforms, data lakes, and lakehouse architectures on AWS, Azure, or Google Cloud Platform.
- Optimize storage costs, compute utilization, query performance, data quality, governance, observability, and platform reliability.
- Develop production-grade Python applications and frameworks, conduct code reviews, and establish engineering best practices.
- Design CI/CD pipelines and automate build, testing, deployment, monitoring, and infrastructure-management processes.
- Manage infrastructure as code and improve deployment reliability and observability.
- Build feature and training data pipelines, operationalize machine learning models, and support AI-enabled data solutions.
- Provide technical leadership, architecture ownership, mentoring, and strategic direction across multiple engineering teams.
- Collaborate with architects, product managers, business stakeholders, and AI teams.
Requirements
- 10+ years of experience in data engineering.
- Strong expertise in Python programming and advanced SQL development and performance tuning.
- Experience with Snowflake or equivalent cloud data warehouse technologies.
- Strong understanding of data modeling, ETL/ELT frameworks, and data governance.
- Experience with Microsoft Azure, AWS, or Google Cloud Platform; Azure is preferred.
- Experience with Azure DevOps, GitHub Actions, Jenkins, GitLab CI/CD, Docker, Kubernetes, and infrastructure as code.
- Exposure to machine learning pipelines, generative AI concepts, LLM integrations, vector databases, RAG architectures, and prompt engineering basics.
- Preferred experience building enterprise-scale data platforms and knowledge of Spark, Databricks, Kafka, streaming technologies, Data Mesh, observability, and monitoring tools.
- Snowflake, Azure, AWS, or GCP certifications are a plus.
- Demonstrated technical leadership, solution architecture, strategic thinking, stakeholder management, problem-solving, mentoring, coaching, and communication skills.
Benefits
- Flexible working environment.
- Volunteer time off.
- LinkedIn Learning.
- Employee Assistance Program (EAP).
Tech Stack
Apache KafkaApache SparkAWSAzureDatabricksDockerGitHub ActionsGitLab CI/CDGoogle Cloud PlatformJenkinsKubernetesPythonSnowflakeSQLTerraform
Categories
Data Engineering