3 months ago
Base Salary
$175k - $225k/yr
Responsibilities
- Architect, develop, and deploy the Big Data environment using technologies such as Kafka, Hadoop, and Dremio.
- Build, deploy, and monitor data processing pipelines using Java, Python, Spark, and Flink.
- Collaborate with development teams on data modeling, data ingestion, and capacity planning.
- Work with users to ensure data integrity and availability.
- Serve as a Big Data subject matter expert and advise users and developers on data-related questions.
- Troubleshoot systems and conduct root-cause analysis.
Requirements
- At least 5 years of experience working in a mature data engineering environment.
- At least 3 years of experience building Kafka streaming applications or maintaining Kafka clusters.
- At least 2 years of experience building applications or pipelines with Big Data backends such as S3, HDFS, Databricks, or Iceberg.
- Experience with Apache Spark, Apache Flink, or similar tools.
- Strong Java, Python, and SQL development skills.
- Experience with common data-science toolkits, especially Python-based toolkits.
- Hands-on experience with Kubernetes and Docker.
- Experience with Prometheus, Grafana, Alert Manager, Alerta, OpsGenie, or similar monitoring tools.
- Strong statistical analysis, troubleshooting, Unix scripting, and root-cause analysis skills.
Benefits
- Full-time, permanent positions are eligible for a discretionary bonus and benefits including paid leave and insurance.
- The role is based on IMC’s Chicago team.
Tech Stack
Apache FlinkApache HadoopApache KafkaApache SparkBashDatabricksDockerGrafanaJavaKubernetesPrometheusPythonSQL
Categories
Data Engineering