14 days ago
Bengaluru, IndiaSenior
Responsibilities
- Design and build scalable AWS data platforms for batch and real-time workloads.
- Develop production-grade Python pipelines, services, and reusable data engineering frameworks.
- Build and maintain batch and streaming pipelines using Apache Kafka, including event processing, schema changes, retries, ordering, duplicate handling, and failure recovery.
- Design and optimize Snowflake data models, queries, warehouses, and workloads alongside relational databases.
- Implement data security, role-based access controls, secure data sharing, governance, traceability, and sensitive-data protections.
- Build pipeline resilience through validation, error handling, retries, monitoring, alerting, and data quality checks.
- Develop automated deployment pipelines and scalable data products and solutions.
- Prepare structured and unstructured data for GenAI use cases, including cleaning, chunking, metadata enrichment, and embedding generation.
- Support OpenAI and retrieval-augmented generation workflows.
- Collaborate with data, advanced analytics, and business teams to gather requirements and deliver fit-for-purpose solutions.
Requirements
- At least 8 years of relevant work experience.
- Experience designing scalable ELT data platforms on AWS for batch and real-time workloads.
- Advanced Python engineering experience, including modular design, error handling, logging, configuration management, testing, and code review.
- Deep Snowflake architecture knowledge and advanced SQL skills, including data modeling, query optimization, and workload management.
- Experience with Oracle, SQL Server, and PostgreSQL, including schema design, transactions, and performance tuning.
- Experience designing, developing, and operating reliable batch and streaming pipelines using Apache Kafka.
- Experience monitoring and optimizing Snowflake queries, warehouse usage, Python workloads, compute resources, cost, throughput, and latency.
- Experience implementing role-based access controls, data protection, secure data-sharing patterns, and governance practices.
- Experience building reliable pipelines with validation, retries, monitoring, alerting, and data quality controls.
- Experience preparing data for GenAI use cases and supporting OpenAI and RAG workflows.
- React and .NET familiarity is advantageous.
- Proven experience in the trading and/or finance industry is preferred.
Benefits
- Competitive salary and benefits.
- Career development opportunities in an international environment.
- Inclusive and accessible recruitment process and workplace, with accommodations available during recruitment or interviews.
- Applications should include a CV and cover letter in English.
Tech Stack
Categories
Data Engineering
About Cargill
Cargill builds and operates global food and agriculture supply chains, selling commodities, food ingredients, animal nutrition, and risk-management services to food manufacturers, retailers, and farmers. It trades and processes grains, oilseeds, cocoa, and proteins, and provides logistics and biofuel solutions. Founded in 1865 and headquartered in Wayzata, Minnesota, Cargill is privately held and family-owned.