8 months ago
London, United KingdomSenior
Responsibilities
- Design and develop scalable data pipelines using Delta Lake, Spark Structured Streaming, and Unity Catalog.
- Build real-time, event-driven solutions using Kafka and Azure Event Hubs.
- Develop CI/CD pipelines with Azure DevOps, Jenkins, or GitHub Actions.
- Collaborate with clients and stakeholders to translate data needs into technical solutions.
- Apply clean-code, data lifecycle optimization, and software engineering best practices.
- Contribute to best practices and continuous improvement across cross-functional engineering teams.
Requirements
- Proven hands-on experience with the Databricks platform and orchestration.
- Strong Python, PySpark, and SQL skills with knowledge of distributed data systems.
- Experience developing full-lifecycle data pipelines across ingestion, transformation, and serving layers.
- Experience with data lakehouse architecture, schema design, and GDPR-compliant solutions.
- Working knowledge of DevOps tools and CI/CD processes.
- Development experience in Scala or Java is a bonus.
- Familiarity with Cloudera, Hadoop, Hive, and the Spark ecosystem is a bonus.
- Understanding of data privacy regulations and experience working with sensitive data is a bonus.
Benefits
- Permanent London role with a hybrid work arrangement.
- Discretionary bonus, competitive pension, health insurance, life insurance, and critical illness cover.
- Mental-health support through CareFirst, Unmind, Aviva consultations, and in-house first aiders.
- Maternity, adoption, shared parental, sickness, pregnancy loss, fertility treatment, menopause, and bereavement leave.
- Eight complimentary backup-care sessions for emergency childcare or elder care.
- Five weeks of annual leave with the option to buy or sell holiday days.
- At least 40 hours of annual training, workshops, certifications, e-learning, and a Business Coach from Day One.
- Online GP services, Gympass (Wellhub), travel insurance, Tastecard, season ticket loans, Cycle to Work, and dental insurance.
Tech Stack
Apache HadoopApache HiveApache KafkaApache SparkAzureDatabricksGitHub ActionsJavaJenkinsPythonScalaSQL
Categories
Data Engineering
About Capco
Capco is a global technology and management consultancy focused on financial services, helping banks, capital markets, insurance, and wealth firms with digital transformation, risk and compliance, and data/AI-enabled modernization. Founded in 1998 and headquartered in London, it operates as a Wipro company. Its business model centers on project-based consulting, managed services, and solution delivery across strategy, engineering, and platforms for regulated institutions.