Software Engineer - Data Engineering
Caterpillar3 days ago
Chennai, India or Bengaluru, IndiaSenior
Responsibilities
- Design, develop, and maintain scalable data pipelines on AWS using S3, Glue, Lambda, Redshift, and EMR.
- Build and optimize Snowflake data warehousing solutions, including performance tuning and data modeling.
- Write reusable Python and SQL code for data transformation and processing.
- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements.
- Monitor, troubleshoot, and improve pipeline performance and reliability.
- Ensure data quality, integrity, and security across the data pipeline.
- Participate in code reviews, architecture discussions, and continuous improvement initiatives.
Requirements
- Require 5–8 years of experience in data engineering or related roles.
- Require strong hands-on experience with AWS cloud services, including data and AI workloads.
- Require deep knowledge of Snowflake architecture, performance tuning, and best practices.
- Require advanced Python and SQL proficiency for data pipelines, transformations, and services.
- Require understanding of graph and vector data modeling and practical applications.
- Require experience with GitHub Actions CI/CD pipelines, cloud-native deployments, and Azure DevOps boards.
- Require a bachelor's or master's degree in Computer Science, Engineering, or a related field.
- Preferred qualifications include NVIDIA ecosystem experience, Milvus or similar vector databases, Neo4j or similar graph databases, AWS Step Functions, data governance and compliance, and real-time processing with Kafka or Spark Streaming.
Benefits
- The position requires working onsite five days per week.
- Relocation is available.
- Visa sponsorship is available for eligible applicants.
Tech Stack
Categories
Data Engineering
About Caterpillar
For 100 years, we’ve been helping customers build a better, more sustainable world. Our innovative products and services, backed by our global dealer network, provide exceptional value that helps customers succeed. With 2024 sales and revenues of $64.8 billion, Caterpillar Inc. is the world’s leading manufacturer of construction and mining equipment, off-highway diesel and natural gas engines, industrial gas turbines and diesel-electric locomotives. Caterpillar does business on every continent, principally operating through three primary segments – Construction Industries, Resource Industries and Energy & Transportation – and provides financing and related services through our Financial Products segment.