14 days ago
Atlanta, GA, USASenior
Responsibilities
- Prepare data infrastructure for efficient data storage and retrieval.
- Examine and resolve data format issues to improve usability and accessibility.
- Develop scalable, sustainable, and robust data products and analytical solutions using advanced engineering and cloud technologies.
- Build and maintain streaming and batch data pipelines for data ingestion, transformation, and movement into data stores.
- Review and optimize existing data systems and architectures.
- Collaborate with data and advanced analytics teams to gather requirements and meet functional and non-functional needs.
- Build prototypes and implement data engineering frameworks and architectures for advanced analytics initiatives.
- Develop automated deployment pipelines with appropriate governance.
- Perform complex data modeling based on datastore technology.
Requirements
- At least 4 years of relevant work experience; the role typically reflects 5 or more years.
- Experience developing data systems on AWS, GCP, or Azure.
- Hands-on experience with modern data architectures, including data lakes, data lakehouses, data hubs, ingestion, governance, modeling, and observability.
- Proficiency with data collection, Kafka, AWS Glue, Iceberg, and Parquet.
- Experience with streaming data pipelines using Kafka and Flink.
- Experience with SQL-based transformation and modeling frameworks and orchestration tools such as dbt, AWS Glue, and Airflow.
- Deep knowledge of SCD and schema evolution concepts.
- Strong experience using Spark for data transformation, streaming, performance tuning, and Spark UI debugging.
- Advanced programming skills in Python, Java, Scala, or similar languages.
- Expert-level SQL proficiency for data manipulation and optimization.
- Experience with DevOps practices, code management, CI/CD, and deployment strategies.
- Strong knowledge of data governance, data quality, privacy, and security considerations.
Tech Stack
Apache AirflowApache FlinkApache KafkaApache SparkAWSAzuredbtGoogle Cloud PlatformJavaPythonScalaSQL
Categories
Data Engineering
About Cargill
Cargill builds and operates global food and agriculture supply chains, selling commodities, food ingredients, animal nutrition, and risk-management services to food manufacturers, retailers, and farmers. It trades and processes grains, oilseeds, cocoa, and proteins, and provides logistics and biofuel solutions. Founded in 1865 and headquartered in Wayzata, Minnesota, Cargill is privately held and family-owned.