Lead Data Engineer – Physical AI Platform
Caterpillar5 days ago
Base Salary
$128k - $209k/yr
Responsibilities
- Collaborate with Principal Software Engineers and Data Architects to define solution architecture.
- Lead the design and optimization of scalable Python data pipelines and microservices for real-time and batch processing.
- Develop cloud-native data ingestion and streaming solutions using AWS services including Kinesis, S3, DynamoDB, and EventBridge.
- Own data integration frameworks and source data pipelines supporting CI Autonomy initiatives.
- Translate business and engineering requirements into scalable data architectures, workflows, mappings, and system designs.
- Establish automated testing, data quality controls, and validation frameworks across distributed data ecosystems.
- Lead monitoring, performance tuning, root-cause analysis, and operational reliability for production data platforms.
Requirements
- Bachelor’s degree in Computer Science, Computer Engineering, or a related field.
- At least 8 years of experience in data engineering or related disciplines with increasing responsibility.
- Extensive experience with modern, large-scale, complex data platforms.
- Strong production experience developing and deploying Python solutions, plus experience with Java and backend systems.
- Experience leading teams building high-throughput, scalable data pipelines.
- Strong hands-on experience with AWS data services, including Kinesis, S3, DynamoDB, and EventBridge.
- Strong SQL skills, including data quality and validation practices.
- Experience with Azure DevOps, Jira, Jenkins, software deployment, microservices, real-time data ingestion, relational databases, and NoSQL databases.
- Experience with monitoring, testing, automation, distributed systems, and ensuring data integrity across streaming platforms.
- Ability to lead complex technical analysis, communicate across teams, document systems, and guide solutions through the full software development lifecycle.
Benefits
- Full-time onsite work in the Irving, TX office five days per week.
- Domestic relocation assistance is available.
- Visa sponsorship is available for eligible applicants.
- Benefits may include medical, dental, vision, paid time off, 401(k), HSA, FSA, wellness programs, employee assistance, discounts, career development, disability and life insurance, parental and adoption benefits, and tuition reimbursement.
- Incentive bonus eligibility may be available subject to plan terms.
About Caterpillar
Caterpillar Inc. designs, manufactures, and services construction and mining equipment, diesel and natural gas engines, industrial gas turbines, and diesel-electric locomotives for infrastructure, resource, and energy customers. The public company, founded in 1925 and headquartered in Irving, Texas, sells primarily through an independent worldwide dealer network and provides equipment financing. It operates through Construction Industries, Resource Industries, and Energy & Transportation segments, with shares traded on the NYSE and Euronext Paris.