Lead Software Engineer - Data Engineering
Caterpillar18 hours ago
Chennai, IndiaStaff+
Responsibilities
- Lead and mentor data engineers and platform developers while driving predictable, high-quality delivery.
- Architect scalable and secure AWS data platforms and define data modeling, governance, and metadata-management practices.
- Design, develop, and optimize batch and near-real-time ingestion pipelines for structured and unstructured data.
- Build and optimize Snowflake-based data platforms for performance and cost.
- Use AWS services including S3, Glue, Lambda, EMR, and Redshift for end-to-end data pipelines.
- Implement CI/CD pipelines with Azure DevOps and Jenkins and ensure scalability, resiliency, and operational readiness.
- Promote software engineering best practices, testing, version control, automation, and reusable ingestion and transformation frameworks.
- Partner with product managers, subject-matter experts, data science and AI teams, and business stakeholders to deliver scalable data solutions.
- Support real-time analytics, edge AI, ML model operationalization, GenAI workflows, and manufacturing data use cases.
Requirements
- 10+ years of experience in data engineering or data platform roles.
- Strong AWS data ecosystem experience with S3, Glue, Lambda, EMR, and Redshift.
- Deep Snowflake expertise including architecture, optimization, and data modeling.
- Strong programming skills in Python and SQL, with extensive ingestion pipeline and ETL/ELT experience.
- Experience with real-time streaming, distributed systems, scalable architectures, and CI/CD tools.
- Experience leading engineering teams and delivering software or platforms at scale.
- Bachelor’s or Master’s degree in Computer Science or Engineering.
- Preferred experience with Kafka, Spark Streaming, GitHub, Jenkins, AWS CloudFormation, Neo4j, Neptune, Milvus, OpenSearch, NVIDIA, RAPIDS, cuDF, cuML, and cuGraph.
- Experience integrating AI/ML pipelines or GenAI workflows and collaborating with data science and AI teams.
Benefits
- The position is based in Chennai at Brigade World Trade Center and requires working onsite five days per week.
- Work-life harmony, earned and medical leave, and relocation assistance.
- Personal and professional development through employee resource groups and global career opportunities.
- Medical, life, and personal accident coverage plus an employee mental wellness assistance program.
- Employee investment plan and an annual incentive bonus plan.
Tech Stack
Categories
Data Engineering
About Caterpillar
Caterpillar Inc. designs, manufactures, and services construction and mining equipment, diesel and natural gas engines, industrial gas turbines, and diesel-electric locomotives for infrastructure, resource, and energy customers. The public company, founded in 1925 and headquartered in Irving, Texas, sells primarily through an independent worldwide dealer network and provides equipment financing. It operates through Construction Industries, Resource Industries, and Energy & Transportation segments, with shares traded on the NYSE and Euronext Paris.