Senior Data Engineer
Intercontinental Exchange1 year ago
Responsibilities
- Design, build, and maintain an on-premises data orchestration platform using open-source tools.
- Create self-service capabilities for building and deploying data pipelines.
- Implement data quality testing frameworks and automated testing across the data lifecycle.
- Establish data engineering practices for version control, deployment, and pipeline operations.
- Build scalable feature engineering pipelines for batch and real-time processing with ML/AI teams.
- Develop reusable data integration patterns and optimize the Kubernetes-based data platform for performance and reliability.
- Mentor junior engineers and promote engineering excellence in data practices.
Requirements
- 5+ years of professional data engineering experience, including at least 2 years working on enterprise-scale data platforms.
- Bachelor's degree in Computer Science or equivalent.
- Deep expertise in workflow orchestration, performance optimization, and operational management.
- Strong understanding of data transformation, testing frameworks, and deployment strategies.
- Experience with stream processing frameworks and technologies.
- Proficiency with SQL and Python for data transformation and pipeline development.
- Familiarity with containerized application deployment.
- Experience implementing data quality frameworks and automated testing for data pipelines.
- Ability to collaborate with data scientists, ML engineers, and business stakeholders.
- Preferred qualifications include experience with self-hosted orchestration platforms, data contracts or schema governance, ML/AI data pipelines, real-time processing, streaming architectures, data modeling, warehouse design, and technical leadership.
Tech Stack
Categories
Data Engineering
About Intercontinental Exchange
Intercontinental Exchange operates global exchanges and clearing houses, offers fixed income and commodities trading platforms, and sells market data, listings, and mortgage technology to financial institutions and lenders. Its revenue comes from transaction and clearing fees, data subscriptions, listings, and software. Founded in 2000 and headquartered in Atlanta, it owns the New York Stock Exchange and ICE Mortgage Technology, and is publicly traded on the NYSE under the ticker ICE.