Infrastructure Validation Engineer
Carbon3 AI4 months ago
London, United KingdomSenior
Responsibilities
- Design and build zero-dependency Python preflight verification tools for distributed infrastructure.
- Develop adaptive test orchestration harnesses that detect environment drift and analyze platform changes.
- Write and maintain Infrastructure as Code for bare-metal and containerized compute environments.
- Automate cluster-level, network, and high-performance storage benchmarking for multi-node GPU systems.
- Validate network fabrics, congestion control, server topology, PCIe links, HBM status, secure boot, and firmware performance.
- Interface with hardware through BMC, IPMI, and Redfish.
- Integrate infrastructure validation suites into CI/CD pipelines and maintain observability pipelines for logs and hardware metrics.
- Partner with platform, network, engineering, product, and datacentre operations teams on root-cause analysis and platform regressions.
Requirements
- Strong proficiency in Python for verification tools, test orchestration, and low-level system checks.
- Deep hands-on experience with Infrastructure as Code for complex bare-metal and containerized environments.
- Proven experience with Kubernetes and enterprise-scale multi-node distributed systems.
- Experience validating server topology, PCIe link speeds, HBM memory, firmware performance, and hardware interfaces.
- Experience integrating automated infrastructure validation into CI/CD pipelines.
- Experience configuring observability pipelines for diagnostic logs and hardware metrics.
- Experience conducting root-cause analysis across platform regressions and hardware-software boundaries.
- Preferred experience validating infrastructure for large-scale model-training workloads, including GPU-to-CPU process affinity and master-to-worker rendezvous routing.
- Preferred understanding of high-throughput network fabrics, traffic patterns, congestion control, GPU clusters, cluster benchmarking, and high-performance storage benchmarking.
- Preferred experience designing intelligent test systems that detect environment drift and analyze large-scale platform changes.
Benefits
- High visibility with leadership, substantial autonomy, and the opportunity to shape operations at scale.
- Mission-driven startup working on critical national infrastructure and renewable-energy-powered AI infrastructure.
- Inclusive and equal-opportunity workplace committed to diversity.
- The role is based within Era4’s UK AI infrastructure and datacentre operations context; no specific remote or office arrangement is stated.
Tech Stack
Categories
About Carbon3 AI
Era4 (formerly Carbon3.ai) builds and operates lower‑carbon, high‑performance AI infrastructure by converting brownfield sites into GPU‑ready data centers. It provides AI compute capacity and Kubernetes‑based platform services for enterprises, ML engineers and data scientists operating in the UK. The company rebranded on 4 March 2026 and develops sites on brownfield land, pairing AI capacity with community regeneration goals.