AI Infrastructure Validation Engineer
Carbon3 AI2 months ago
London, United KingdomSenior
Responsibilities
- Design and build zero-dependency Python preflight verification tools for multi-node distributed systems.
- Create adaptive test orchestration harnesses that detect environment drift and analyze platform changes.
- Write and maintain Infrastructure as Code for provisioning, configuring, testing, and tearing down bare-metal and containerized compute environments.
- Automate cluster-level and high-performance storage benchmarking to validate node-to-node throughput and network performance.
- Validate server topology, PCIe link speeds, HBM memory, secure boot, firmware performance, and hardware interfaces through BMC, IPMI, and Redfish.
- Integrate infrastructure validation suites into CI/CD pipelines and maintain observability pipelines for diagnostic logs and hardware metrics.
- Lead root-cause analysis with Platform, Network Engineering, and Datacentre Operations teams on platform regressions and interconnect bottlenecks.
Requirements
- Strong proficiency in Python for verification tools, automated test orchestration, and low-level system checks.
- Deep hands-on experience writing and maintaining Infrastructure as Code for bare-metal and containerized compute environments.
- Proven experience with Kubernetes and enterprise-scale, multi-node distributed systems.
- Experience validating server topology, PCIe links, HBM memory, firmware performance, and hardware interfaces using BMC, IPMI, or Redfish.
- Demonstrated ability to integrate automated infrastructure validation into CI/CD pipelines and configure observability pipelines.
- Experience conducting root-cause analysis across engineering, product, and datacentre operations.
- Preferred experience validating infrastructure for large-scale model training, including GPU-to-CPU process affinity and master-to-worker rendezvous routing.
- Preferred understanding of high-throughput network fabrics, traffic patterns, congestion control, multi-node GPU clusters, and performance benchmarking.
- Preferred experience designing intelligent test systems that autonomously detect environment drift and analyze large-scale platform changes.
Benefits
- Permanent position; contractors are also considered and should provide a day rate in the application.
- High visibility with leadership, significant autonomy, and the opportunity to shape operations at scale.
- Mission-driven start-up focused on renewable-energy-powered AI infrastructure and inclusive workplace practices.
Tech Stack
Categories
About Carbon3 AI
Era4 (formerly Carbon3.ai) builds and operates lower‑carbon, high‑performance AI infrastructure by converting brownfield sites into GPU‑ready data centers. It provides AI compute capacity and Kubernetes‑based platform services for enterprises, ML engineers and data scientists operating in the UK. The company rebranded on 4 March 2026 and develops sites on brownfield land, pairing AI capacity with community regeneration goals.