DevOps Engineer - AI Runtime Services
Demandbase2 months ago
Hyderābād, IndiaSenior
Responsibilities
- Own the LLM gateway, including model-provider keys, routing, rate limits, failover, and provider quotas.
- Own runtime reliability through SLOs, on-call, incident response, and postmortems.
- Build cost attribution, budgets, quotas, and showback controls for teams and models.
- Develop LLM observability covering traces, spans, prompt and response capture, ingest, and billing visibility.
- Build shared evaluation and experimentation infrastructure, including harnesses, datasets, and scoring systems.
- Implement response and semantic caching, input and output safety, PII handling, and policy enforcement.
- Operate and improve the AI runtime and internal tooling in production across product-facing and internal systems.
Requirements
- Strong production infrastructure experience with Kubernetes, AWS, Terraform, and GitOps.
- Advanced Python experience sufficient to own services and tooling.
- Real on-call and incident-response experience, including leading incidents and implementing post-incident improvements.
- Experience instrumenting systems and operating metrics and logging backends with effective alerting.
- Hands-on experience building with LLM systems, such as an agent, RAG pipeline, or internal tool, and using evals or experiments to assess quality.
- Familiarity with tokens, context windows, prompt and response tracing, and the importance of evaluation suites.
- Experience with an LLM gateway or proxy such as LiteLLM is preferred.
- Experience with LLM observability tools such as Datadog LLM Observability, LangSmith, Braintrust, or Arize is preferred.
- Experience with eval frameworks, LLM-as-judge scoring, FinOps, cost attribution, quota design, guardrails, PII detection, or content filtering is preferred.
- Experience with GCP alongside AWS is preferred.
Benefits
- Group Medical, Personal Accident, and Term Life Insurance
- Dental, vision, and OPD coverage
- Mental health support
- Fitness benefit
- Car lease policy
- Gratuity
Tech Stack
Categories
DevOpsSite Reliability
About Demandbase
Demandbase is the only pipeline AI platform that empowers GTM teams to automate growth at scale. With a unified view of data, insights, actions, and outcomes, B2B enterprises can seamlessly align and execute their account-based GTM strategies with confidence. Thousands of businesses trust Demandbase to maximize revenue, minimize waste, and consolidate their data and tech stacks – all in one platform. For more information about how Demandbase can help you accelerate your pipeline at scale, visit www.demandbase.com