Software Engineer - Data Engineering
Caterpillar1 day ago
Chennai, India or Bengaluru, IndiaSenior
Responsibilities
- Design, develop, and maintain scalable data pipelines on AWS using S3, Glue, Lambda, Redshift, and EMR.
- Build and optimize Snowflake data warehousing solutions through performance tuning and data modeling.
- Write reusable Python and SQL code for data transformation and processing.
- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements.
- Monitor, troubleshoot, and improve pipeline performance and reliability.
- Ensure data quality, integrity, and security across the data pipeline.
- Participate in code reviews, architecture discussions, and continuous improvement initiatives.
Requirements
- 5-8 years of experience in data engineering or related roles.
- Strong hands-on experience with AWS cloud services, including data and AI workloads.
- Deep understanding of Snowflake architecture, performance tuning, and best practices.
- Advanced proficiency in Python and SQL for data pipelines, transformations, and services.
- Strong understanding of graph and vector data modeling concepts and practical applications.
- Experience developing and operating CI/CD pipelines with GitHub Actions and cloud-native deployments.
- Experience using Azure DevOps boards for backlog management in Agile environments.
- Strong analytical, problem-solving, communication, and collaboration abilities.
- Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
- Knowledge of the NVIDIA ecosystem and its applications in data and AI is preferred.
- Hands-on experience with Milvus or similar vector databases and Neo4j or similar graph databases is preferred.
- Experience with orchestration tools such as AWS Step Functions is preferred.
- Familiarity with data governance and compliance practices is preferred.
- Exposure to real-time data processing frameworks such as Kafka and Spark Streaming is preferred.
Benefits
- The posting identifies Caterpillar as an Equal Opportunity Employer.
- The posting dates are October 28, 2026 through November 3, 2026.
Tech Stack
Categories
Data Engineering
About Caterpillar
Caterpillar Inc. designs, manufactures, and services construction and mining equipment, diesel and natural gas engines, industrial gas turbines, and diesel-electric locomotives for infrastructure, resource, and energy customers. The public company, founded in 1925 and headquartered in Irving, Texas, sells primarily through an independent worldwide dealer network and provides equipment financing. It operates through Construction Industries, Resource Industries, and Energy & Transportation segments, with shares traded on the NYSE and Euronext Paris.