Senior Software Engineer - Data Engineering
Caterpillar3 days ago
Chennai, India or Bengaluru, IndiaSenior
Responsibilities
- Design, develop, and maintain scalable data pipelines on AWS using S3, Glue, Lambda, Redshift, and EMR.
- Build and optimize Snowflake data warehousing solutions through performance tuning and data modeling.
- Write reusable Python and SQL for data transformation and processing.
- Collaborate with data scientists, analysts, and business stakeholders to define data requirements.
- Integrate vector databases with LLM-based applications and AI workflows.
- Monitor, troubleshoot, and improve data pipeline performance and reliability.
- Ensure data quality, integrity, and security across the pipeline lifecycle.
- Participate in code reviews, architecture discussions, and continuous improvement initiatives.
Requirements
- Requires 8+ years of experience in data engineering or related roles.
- Requires strong hands-on experience with AWS cloud services and data and AI workloads.
- Requires deep knowledge of Snowflake architecture, performance tuning, and best practices.
- Requires advanced Python and SQL proficiency for data pipelines, transformations, and services.
- Requires hands-on experience with graph databases such as Neo4j or Neptune and vector databases such as Milvus or Amazon OpenSearch.
- Requires experience with Git and Git workflows.
- Requires experience using Azure DevOps boards for backlog management in Agile environments.
- Requires strong analytical, problem-solving, communication, and collaboration skills.
- Requires a bachelor's or master's degree in Computer Science, Engineering, or a related field.
- Preferred qualifications include NVIDIA ecosystem knowledge, RAPIDS or CUDA exposure, Milvus expertise, Neo4j knowledge graph ingestion experience, AWS Step Functions, data governance and compliance experience, and Kafka or Spark Streaming exposure.
Benefits
- The position requires working onsite five days per week.
- Relocation is available for this position.
Tech Stack
Categories
Data Engineering
About Caterpillar
For 100 years, we’ve been helping customers build a better, more sustainable world. Our innovative products and services, backed by our global dealer network, provide exceptional value that helps customers succeed. With 2024 sales and revenues of $64.8 billion, Caterpillar Inc. is the world’s leading manufacturer of construction and mining equipment, off-highway diesel and natural gas engines, industrial gas turbines and diesel-electric locomotives. Caterpillar does business on every continent, principally operating through three primary segments – Construction Industries, Resource Industries and Energy & Transportation – and provides financing and related services through our Financial Products segment.