Senior Data Engineer
Intercontinental Exchange2 months ago
Responsibilities
- Design, build, and maintain an on-premises data orchestration platform using open-source tools.
- Create self-service capabilities for teams to build and deploy data pipelines.
- Implement data quality testing frameworks and automated testing across the data lifecycle.
- Establish data engineering practices for version control, CI/CD, and pipeline deployment.
- Build scalable feature engineering pipelines for batch and real-time processing with ML/AI teams.
- Develop reusable data integration patterns for organization-wide use.
- Optimize the Kubernetes-based data platform for performance and reliability with infrastructure teams.
- Mentor junior engineers and advocate for engineering excellence in data practices.
Requirements
- Bachelor’s degree in Computer Science, Computer Engineering, Data Science, Mathematics, Engineering, or a related discipline.
- At least 5 years of professional data engineering experience, including at least 2 years working on enterprise-scale data platforms.
- Deep expertise in workflow orchestration, performance optimization, and operational management.
- Strong understanding of data transformation techniques, testing frameworks, and deployment strategies.
- Experience with stream processing frameworks and technologies.
- Proficiency with SQL and Python for data transformation and pipeline development.
- Familiarity with containerized application deployment.
- Experience implementing data quality frameworks and automated testing for data pipelines.
- Ability to collaborate cross-functionally with data scientists, ML engineers, and business stakeholders.
- Preferred experience with self-hosted data orchestration platforms, data contracts or schema governance, ML/AI data pipelines, feature engineering, real-time processing, streaming architectures, data modeling, warehouse design, and technical leadership.
Benefits
- Onsite work arrangement, as indicated by the posting.
Tech Stack
Categories
Data Engineering
About Intercontinental Exchange
Intercontinental Exchange operates global exchanges and clearing houses, offers fixed income and commodities trading platforms, and sells market data, listings, and mortgage technology to financial institutions and lenders. Its revenue comes from transaction and clearing fees, data subscriptions, listings, and software. Founded in 2000 and headquartered in Atlanta, it owns the New York Stock Exchange and ICE Mortgage Technology, and is publicly traded on the NYSE under the ticker ICE.