Data Engineer
Arrow Global Group9 months ago
Manchester, United KingdomMid Level
Responsibilities
- Build and maintain ingestion and transformation pipelines using Microsoft Fabric, Data Pipelines, PySpark notebooks, and Dataflows Gen2.
- Onboard source systems through reusable ingestion patterns including mirroring, change data capture, file loads, and API loads.
- Write and tune PySpark and T-SQL for Lakehouse and Warehouse workloads, including fact and dimension models.
- Implement data quality rules and validation pipelines and investigate reconciliation and data-quality issues.
- Use Git, Azure DevOps, pull requests, peer review, and CI/CD promotion across development, UAT, and production.
- Contribute to Terraform infrastructure-as-code and SQL project or DACPAC orchestration configuration.
- Collaborate with Data Stewards, Data Analysts, and Data Architects to implement mappings, business rules, and transformations.
- Maintain source-to-target mappings, transformation documentation, and operational runbooks.
- Monitor pipeline runs, triage failures, and support the data platform day to day.
- Prepare trusted datasets and Power BI semantic models for portfolio analytics, asset strategy, finance, and risk.
Requirements
- Experience building data pipelines as a Data Engineer or in a similar role.
- Strong Python, PySpark, and SQL skills.
- Hands-on Microsoft Fabric experience, or experience with Databricks, Azure Synapse, or Azure Data Factory and willingness to move to Fabric.
- Comfort with Git, pull requests, and CI/CD; Terraform or other infrastructure-as-code experience is a plus.
- Understanding of dimensional or medallion data modeling and data quality practices.
- Strong analytical, verbal, and written communication skills and the ability to work collaboratively.
- A degree in Computer Science, Information Systems, Data Science, or a related field is preferred.
- Financial data experience in areas such as loan servicing, transactions, reference data, IBOR, or ABOR, or experience in Real Estate, Credit, Banking, or NPL Asset Management, is desirable.
- Microsoft certification such as DP-700 Fabric Data Engineer is a plus.