Senior Data Engineer
Caterpillar19 hours ago
Bengaluru, IndiaSenior
Responsibilities
- Design, develop, and maintain scalable batch and near-real-time data pipelines on AWS.
- Build connectors and integrate data from APIs, databases, files, and multiple other sources into Coveo and downstream systems.
- Own data quality validation covering completeness, freshness, consistency, accuracy, schema validation, and anomaly detection.
- Implement automated data quality checks, alerts, and monitoring for pipeline failures and data issues.
- Collaborate with search and platform teams to ensure high-quality data is indexed in Coveo.
- Analyze search-indexed datasets to identify quality issues and improve relevance, accuracy, and search performance.
- Contribute to CI/CD pipelines while following enterprise SDLC and security standards.
- Document pipelines, data contracts, and quality rules.
Requirements
- Strong experience as an AWS Data Engineer.
- Hands-on experience with AWS Glue, Bedrock, Lambda, Step Functions, S3, and SNS/SQS or similar services.
- Proficiency in Python and/or PySpark for data engineering and validation logic.
- Strong understanding of data quality concepts, frameworks, and best practices.
- Experience integrating data from APIs, databases, and files.
- Knowledge of ETL/ELT patterns and data modeling fundamentals.
- Experience with logging, monitoring, and alerting for data pipelines.
- Familiarity with CI/CD practices and Git version control.
- Experience with Coveo sources, connectors, indexing, or ingestion pipelines is preferred.
- Prior experience supporting search platforms or content indexing pipelines is preferred.
- Exposure to data quality frameworks or rule-based validation approaches is preferred.
- Knowledge of Snowflake and experience in large enterprise or multi-team environments are preferred.
- Strong problem-solving, debugging, communication, cross-functional collaboration, ownership, and attention-to-detail skills.
About Caterpillar
Caterpillar Inc. designs, manufactures, and services construction and mining equipment, diesel and natural gas engines, industrial gas turbines, and diesel-electric locomotives for infrastructure, resource, and energy customers. The public company, founded in 1925 and headquartered in Irving, Texas, sells primarily through an independent worldwide dealer network and provides equipment financing. It operates through Construction Industries, Resource Industries, and Energy & Transportation segments, with shares traded on the NYSE and Euronext Paris.