13 days ago
Remote, United StatesSenior
Base Salary
$80k - $137k/yr
Responsibilities
- Design, develop, enhance, and own scalable data pipelines, data processing systems, and data products.
- Build reusable ingestion, transformation, storage, data modeling, and schema patterns for product and analytical needs.
- Integrate data from APIs, databases, event streams, and third-party systems.
- Apply testing, version control, CI/CD, automated deployment, validation, monitoring, and observability practices.
- Optimize data workflows, SQL queries, pipelines, databases, and storage for performance, scalability, cost efficiency, and resilience.
- Support data quality, integrity, governance, metadata management, lineage, privacy, security, and regulatory compliance.
- Collaborate with product managers, engineers, and analysts in planning, standups, retrospectives, and other product ceremonies.
- Evaluate new technologies and patterns, document technical components and data flows, and mentor less experienced engineers.
Requirements
- Strong proficiency developing scalable data pipelines with PySpark and Python.
- Hands-on experience with modern data engineering frameworks and tools such as Apache Spark, Kafka, Flink, and dbt or equivalent technologies.
- Strong knowledge of SQL, NoSQL databases, data modeling, and data architecture principles.
- Experience designing batch and/or real-time streaming data solutions.
- Hands-on experience with Microsoft Fabric Notebooks for developing, debugging, testing, and operationalizing data engineering workflows.
- Experience with Docker and Kubernetes.
- Experience implementing monitoring and observability solutions; Datadog experience is preferred.
- Preferred experience with Azure, AWS, Microsoft Fabric Lakehouse, Warehouse, Data Pipelines, Dataflows Gen2, and Semantic Models.
- Familiarity with CI/CD tools such as Jenkins, GitHub Actions, and Azure DevOps.
- Familiarity with generative and agentic AI tools for engineering productivity.
- Strong collaboration, communication, analytical, troubleshooting, and problem-solving skills.
Benefits
- Allstate provides a technology setup including a laptop, monitors, headset, keyboard, and mouse.
- Employees eligible to work from home receive a monthly connectivity reimbursement for internet costs.
- Work-from-home employees must have a dedicated private workspace, appropriate desk and seating, and reliable internet with minimum speeds of 50 MB download and 5 MB upload.
- Candidates must complete a background investigation if offered the position.
Tech Stack
Apache FlinkApache KafkaApache SparkAWSAzureDatadogdbtDockerGitHub ActionsJenkinsKubernetesPythonSQL
Categories
Data Engineering
About Allstate
Allstate is a major U.S. personal-lines insurer that sells auto, homeowners, renters, and life policies to consumers through agents and direct online channels. It earns premiums and service fees across insurance and protection products, including telematics-based programs and device/identity protection. Founded in 1931 and headquartered in Northbrook, Illinois, Allstate is publicly traded on the NYSE under the ticker ALL.