Software Engineer - Data Engineering
Caterpillar17 hours ago
Chennai, India or Bengaluru, IndiaSenior / Staff+
H1B Sponsor
Responsibilities
- Design, develop, and maintain scalable data pipelines on AWS using S3, Glue, Lambda, Redshift, and EMR.
- Build and optimize Snowflake data warehousing solutions, including data modeling and performance tuning.
- Write efficient, reusable Python and SQL code for data transformation and processing.
- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements.
- Monitor, troubleshoot, and improve pipeline performance and reliability.
- Ensure data quality, integrity, and security throughout the data pipeline.
- Participate in code reviews, architecture discussions, and continuous improvement initiatives.
Requirements
- Require 5–8 years of experience in data engineering or related roles.
- Require strong hands-on experience with AWS cloud services and deep knowledge of Snowflake architecture, performance tuning, and best practices.
- Require advanced Python and SQL proficiency for data pipelines, transformations, and services.
- Require understanding of graph and vector data modeling concepts and practical applications.
- Require experience developing and operating CI/CD pipelines with GitHub Actions and cloud-native deployments.
- Require experience using Azure DevOps boards for backlog management in Agile environments.
- Require a bachelor’s or master’s degree in Computer Science, Engineering, or a related field.
- Preferred experience includes the NVIDIA ecosystem, Milvus or similar vector databases, and Neo4j or similar graph databases.
- Preferred experience includes AWS Step Functions, data governance and compliance practices, and real-time processing frameworks such as Kafka and Spark Streaming.
Benefits
- The position requires onsite work five days per week.
- Relocation assistance is available.
Tech Stack
Categories
Data Engineering
About Caterpillar
For 100 years, we’ve been helping customers build a better, more sustainable world. Our innovative products and services, backed by our global dealer network, provide exceptional value that helps customers succeed. With 2024 sales and revenues of $64.8 billion, Caterpillar Inc. is the world’s leading manufacturer of construction and mining equipment, off-highway diesel and natural gas engines, industrial gas turbines and diesel-electric locomotives. Caterpillar does business on every continent, principally operating through three primary segments – Construction Industries, Resource Industries and Energy & Transportation – and provides financing and related services through our Financial Products segment.