Lead Software Engineer- Data Engineer
Caterpillar21 hours ago
Chennai, India or Bengaluru, IndiaStaff+
Responsibilities
- Lead and mentor a team of data engineers and platform developers.
- Architect scalable and secure AWS-based data platforms.
- Design and develop batch and near-real-time ingestion pipelines for structured and unstructured data.
- Build and optimize Snowflake data platforms, including data modeling, governance, metadata management, performance, and cost optimization.
- Use AWS services to implement end-to-end data pipelines.
- Establish engineering best practices, code quality, testing, version control, CI/CD standards, and reusable ingestion and transformation frameworks.
- Drive Agile delivery, automation, continuous improvement, scalability, resiliency, and operational readiness.
- Partner with product managers, subject matter experts, data science and AI teams, and business stakeholders to deliver scalable data solutions.
- Operationalize ML models and analytics workflows and support real-time analytics, edge AI, and AI-enabled data engineering use cases.
Requirements
- 10+ years of experience in data engineering or data platform roles.
- Strong experience with the AWS data ecosystem, including S3, Glue, Lambda, EMR, and Redshift.
- Deep expertise in Snowflake architecture, optimization, and data modeling.
- Strong programming skills in Python and SQL.
- Extensive experience with data ingestion pipelines and ETL/ELT frameworks.
- Experience with real-time streaming technologies such as Kafka and Spark Streaming.
- Experience with CI/CD and engineering tools including GitHub, Jenkins, AWS CloudFormation, and Git.
- Understanding of distributed systems, scalable architectures, software engineering principles, testing, and design patterns.
- Experience working with Agile teams and leading engineering teams at scale.
- Bachelor’s or master’s degree in Computer Science or Engineering.
- Preferred experience with graph databases such as Neo4j and Neptune; vector databases such as Milvus and OpenSearch; and the NVIDIA ecosystem, including RAPIDS, cuDF, cuML, and cuGraph.
- Preferred experience integrating AI/ML pipelines or GenAI workflows.
Benefits
- The position requires working onsite five days per week.
- Relocation is available.
- The posting is open from November 4, 2026 through November 12, 2026.
Tech Stack
Categories
Data Engineering
About Caterpillar
Caterpillar Inc. designs, manufactures, and services construction and mining equipment, diesel and natural gas engines, industrial gas turbines, and diesel-electric locomotives for infrastructure, resource, and energy customers. The public company, founded in 1925 and headquartered in Irving, Texas, sells primarily through an independent worldwide dealer network and provides equipment financing. It operates through Construction Industries, Resource Industries, and Energy & Transportation segments, with shares traded on the NYSE and Euronext Paris.