Senior Data Engineer (Information Systems Analyst III Opt. S)
State of Illinois3 days ago
Responsibilities
- Lead the design, development, and optimization of enterprise ETL and ELT pipelines, distributed processing workloads, and semantic data models.
- Architect and govern enterprise data lakes, lakehouses, warehouses, storage tiering, catalog layers, and data asset observability.
- Design and maintain CI/CD and DataOps automation for data engineering workloads, including code promotion and production deployment processes.
- Implement and govern role-based access control and secure, auditable data access aligned with enterprise and regulatory requirements.
- Engineer and optimize workloads across relational, non-relational, document, key-value, graph, cloud, and emerging data platforms.
- Optimize SQL performance, Spark execution, batch, micro-batch, and streaming workloads, including execution plans, partitioning, memory management, and autoscaling.
- Provide technical leadership, architecture guidance, peer code review, mentorship, and advisory support to cross-functional teams, vendors, and executive stakeholders.
- Maintain data quality, lineage, schema evolution, anomaly detection, pipeline alerting, and operational reliability standards.
Requirements
- Four years of college coursework in Computer Science, Computer Engineering, Information Systems, Data Engineering, Software Engineering, Management Information Science, or a closely related field, or equivalent knowledge, skill, and mental development.
- At least five years of professional experience in a related information technology field, including data engineering, software engineering, database administration, business intelligence development, data architecture, or systems integration.
- At least five years of professional experience designing and optimizing enterprise-scale data pipelines, fault-tolerant ETL/ELT patterns, distributed processing systems, and SQL performance across relational and cloud data systems.
- At least five years of professional experience with distributed data processing and Spark execution engine optimization across large-scale batch, micro-batch, and streaming workloads.
- At least five years of professional experience designing enterprise data platform and lakehouse architectures, including multi-zone lake patterns, open table formats, schema evolution, and storage, compute, and catalog integration.
- At least four years of professional experience engineering reusable data access layers and gateway patterns for governed enterprise interoperability.
- Preferred experience includes implementing role-based access control, data observability, data quality monitoring, DataOps, CI/CD, and infrastructure as code for enterprise data workloads.
- Ability to mentor junior and mid-level engineers, perform expert peer code reviews, analyze data, recommend technical solutions, and communicate technical concepts to technical and non-technical audiences.
- Must be authorized to accept a permanent full-time position with the State of Illinois, pass an agency-required background check, travel occasionally, work outside normal business hours when needed, and meet stated physical and ethics requirements.
Benefits
- Competitive health, life, dental, and vision insurance.
- Hybrid work schedule with one to five days per week in the Chicago office depending on operational needs; normal hours are Monday through Friday, 8:30 AM to 5:00 PM, with hours potentially varying.
- 10 to 25 days of paid vacation annually, including 10 days during the first year of state employment.
- 12 days of paid sick time annually, three paid personal business days, and 13 to 14 paid holidays.
- 12 weeks of paid parental leave.
- State Employees Retirement System pension plan and voluntary deferred compensation program.
- Optional pre-tax Medical Care Assistance Plan and Dependent Care Assistance Plan.
- Tuition reimbursement and eligibility for the Federal Public Service Loan Forgiveness Program.
Tech Stack
Amazon DynamoDBApache SparkCouchbaseDatabricksdbtGitHub ActionsIBM DB2Microsoft SQL ServerMongoDBNeo4jRedisSnowflakeSQLTerraform
Categories
Data Engineering
About State of Illinois
State of Illinois is the state government providing public services, regulation, and infrastructure for residents and businesses. It operates departments and programs spanning public safety, health and human services, transportation, education support, revenue, and environmental protection, funded by the state budget and taxes. Established in 1818, the government is headquartered in Springfield with major offices statewide, including Chicago, and hires across civil service, technical, and facility operations roles.