Senior Data Platform Engineer
Octopus Energy Group1 month ago
London, United KingdomSenior
Responsibilities
- Build and maintain Terraform Infrastructure-as-Code across development, test, and production environments.
- Strengthen GitHub Actions CI/CD with repeatable deployments, environment promotion, automated rollbacks, and release controls.
- Design and implement federated identity architecture across Okta, Entra ID, and ZScaler.
- Build identity, token exchange, permission propagation, SCIM provisioning, and auditable access lifecycle models for human users and non-human AI agents.
- Harden AI gateway and agent orchestration layers using MCP patterns, including authentication, routing, rate limiting, token budgets, policy-as-code, and action logging.
- Implement platform, tool, and agent kill switches plus safe-testing and dry-run modes.
- Implement AI and gateway observability covering prompt logging, output monitoring, quality scoring, drift detection, latency, errors, and model routing.
- Establish token budgets, cost alerts, and usage and cost attribution by team and use case.
- Deploy vector database and retrieval infrastructure, including Databricks AI Search, for Document AI and RAG use cases.
- Provide documentation, reusable patterns, and self-service onboarding guidance for delivery squads.
Requirements
- Proven experience designing and implementing enterprise federated identity, token exchange, permission propagation, SCIM provisioning, and access lifecycle management.
- Strong hands-on experience with Terraform and production CI/CD pipelines, including environment promotion, rollback, and release controls.
- Experience hardening API gateway patterns involving authentication, routing, rate limiting, and access policy enforcement.
- Technical comfort working across Azure, AWS, or Databricks platform ecosystems.
- Ability to work autonomously in a lean team, translate architectural specifications into practical code, and communicate trade-offs clearly.
- Experience with AI gateways, LLM proxies, or model-routing patterns is advantageous.
- Exposure to AI observability tools, vector databases, Databricks AI Search, RAG pipelines, or monitoring non-deterministic systems is advantageous.
- Hands-on experience with policy-as-code, action logging, or AI kill switches is advantageous.
- Direct experience with Okta, ZScaler, Entra ID, Unity Catalog, or M365 migration environments is advantageous.
- Background in regulated or high-governance industries such as Financial Services, Energy, Infrastructure Investment, Fintech, or Enterprise SaaS is advantageous.
- Familiarity with FinOps practices including token budgeting, usage tracking, and cost attribution is advantageous.
Benefits
- Flexible salary discussed with recruiters rather than advertised.
- Autonomous culture with opportunities to learn, decide, and build quickly alongside co-owners.
- Recruitment process usually takes up to four weeks with flexibility where possible.
- Interview accommodations and customized interview support are available upon request.
- Equal opportunity employer committed to an inclusive and fair work environment.