2 hours ago
Bengaluru, IndiaStaff+
Responsibilities
- Lead the design, development, and maintenance of data engineering infrastructure.
- Architect scalable batch and real-time data pipelines and end-to-end data solutions.
- Migrate on-premises applications, multi-terabyte databases, and data warehouses to cloud-based platforms.
- Design conceptual and logical data models, source-to-target mappings, transformations, flowcharts, and data lineage.
- Create data architecture, engineering, and migration patterns across relational, NoSQL, analytical, big data, and warehouse technologies.
- Establish coding standards, engineering best practices, technical documentation, and data governance practices.
- Mentor junior data engineers and provide technical guidance to data and analytics engineering teams.
- Collaborate with application architects, data scientists, analysts, and business stakeholders on data-intensive applications and requirements.
- Optimize data models, database designs, pipelines, and data warehouses for performance and reliability.
- Implement data quality controls, monitoring systems, security controls, compliance, and governance standards.
- Evaluate and recommend data technologies and tools and manage cloud-based data solutions.
- Translate business requirements into technical solutions and improve data quality and analytics processes.
Requirements
- Bachelor's degree in Computer Science, Engineering, or a related technical field; a master's degree is preferred.
- 10+ years of experience in data engineering roles.
- 2+ years of experience in a technical leadership position.
- Hands-on technical involvement with real-time, batch, and NoSQL/SQL systems.
- Proficiency in data warehousing, data modeling, ETL design, and optimization.
- Deep understanding of data enrichment, transformation, security, movement, data architecture, golden records, and data integrity.
- Experience with big data technologies such as Hadoop, Spark, and Kafka.
- Experience with data orchestration tools such as Airflow and Luigi.
- Familiarity with AWS Database Migration Service and Server Migration Service.
- Experience leading data or analytics engineering teams.
- Experience with agile development practices and strong problem-solving and debugging skills.
- Strong communication skills for presenting insights to stakeholders.
- Product-based company experience is mandatory.
Tech Stack
Amazon DynamoDBAmazon RedshiftApache AirflowApache CassandraApache HadoopApache HiveApache KafkaApache SparkAWSAzureDatabricksDatadogdbtDockerGitGitHub ActionsGitLab CI/CDGoogle BigQueryGoogle Cloud PlatformGrafanaInfluxDBInformaticaJavaJenkinsKubernetesMicrosoft SQL ServerMongoDBMySQLPostgreSQLPythonScalaSnowflakeSQLTalendTerraform
Categories
Data Engineering
