Google Cloud Platform Data Engineer
PA Consulting4 months ago
Bristol, United KingdomStaff+
Responsibilities
- Develop production-ready data processing jobs and automated pipelines using Google Cloud services such as Dataflow, Dataproc, BigQuery, and Cloud Composer.
- Design end-to-end batch and streaming data solutions and contribute to broader architecture discussions.
- Own and improve the team’s development processes, architecture standards, code quality practices, deployments, specifications, acceptance criteria, estimation, planning, and documentation.
- Define and evolve data engineering standards and practices across the organization.
- Lead technical discussions with client stakeholders and secure alignment on solution designs.
- Coach and mentor engineers while building team expertise.
- Develop data quality alerting and data quarantine solutions for trusted downstream datasets.
- Implement automated testing and deployment pipelines using Git-based branching and code control.
- Translate business requirements into technical specifications and data solution designs.
Requirements
- Expert experience delivering and deploying production-ready data processing solutions using BigQuery, Pub/Sub, Dataflow, and Dataproc.
- Experience developing end-to-end solutions with batch and streaming frameworks such as Apache Spark and Apache Beam.
- Expert understanding of relational and non-relational storage, document stores, row-based and columnar stores, data warehouses, and data lakes.
- Expert understanding of event-driven architectures, ETL/ELT, stream processing, and data visualization approaches.
- Experience working with business owners to translate requirements into technical specifications and solution designs.
- Experience with Cloud Data Catalog, Collibra, and Dataplex for metadata management and data governance.
- Experience developing cloud-native solutions on Google Cloud Platform.
- Experience implementing CI/CD pipelines with source control, automated tests, and automated deployments.
- Experience working in Agile teams using Scrum or Kanban.
- Preferred experience migrating enterprise-scale data platforms, including Hadoop and traditional data warehouses.
- Preferred understanding of the machine learning model development lifecycle, feature engineering, training, and testing.
- Preferred hands-on experience with Kafka.
- Preferred experience as a DBA or developer using PostgreSQL, MySQL, Oracle, or SQL Server.
- Experience designing data applications for non-functional requirements such as performance and availability.
- Strong communication, troubleshooting, problem-solving, stakeholder influence, and mentoring skills.
Benefits
- Private medical insurance.
- Interest-free season ticket loan.
- 25 days of annual leave with the opportunity to buy 5 additional days.
- Company pension scheme.
- Annual performance-based bonus.
- Life and income protection insurance.
- Tax-efficient benefits including cycle to work, give as you earn, and childcare benefits.
- Voluntary benefits including dental, critical illness, and spouse/partner life assurance.
- Inclusive and supportive workplace culture with opportunities for professional growth.
Tech Stack
Apache BeamApache HadoopApache KafkaApache SparkGitGoogle BigQueryGoogle CloudGoogle Cloud PlatformMicrosoft SQL ServerMySQLPostgreSQL
Categories
Data Engineering
About PA Consulting
PA Consulting is a management and technology consultancy that designs and delivers strategy, digital systems, and physical products for public- and private-sector clients. It works across defense and security, government, health and life sciences, financial services, energy and utilities, consumer, and transport markets. Founded in 1943 and headquartered in London, it operates globally and is majority-owned by Jacobs.