DevOps Engineer - AI Runtime Services
Demandbase3 months ago
Hyderābād, IndiaSenior
Responsibilities
- Own the LLM gateway, including model-provider keys, routing, rate limits, failover, and provider quotas.
- Own runtime reliability through SLOs, on-call, incident response, and postmortems.
- Build cost attribution, budgets, quotas, and showback controls for teams and models.
- Develop LLM observability covering traces, spans, prompt and response capture, ingest, and billing visibility.
- Build shared evaluation and experimentation infrastructure, including harnesses, datasets, and scoring systems.
- Implement response and semantic caching, input and output safety, PII handling, and policy enforcement.
- Operate and improve the AI runtime and internal tooling in production across product-facing and internal systems.
Requirements
- Strong production infrastructure experience with Kubernetes, AWS, Terraform, and GitOps.
- Advanced Python experience sufficient to own services and tooling.
- Real on-call and incident-response experience, including leading incidents and implementing post-incident improvements.
- Experience instrumenting systems and operating metrics and logging backends with effective alerting.
- Hands-on experience building with LLM systems, such as an agent, RAG pipeline, or internal tool, and using evals or experiments to assess quality.
- Familiarity with tokens, context windows, prompt and response tracing, and the importance of evaluation suites.
- Experience with an LLM gateway or proxy such as LiteLLM is preferred.
- Experience with LLM observability tools such as Datadog LLM Observability, LangSmith, Braintrust, or Arize is preferred.
- Experience with eval frameworks, LLM-as-judge scoring, FinOps, cost attribution, quota design, guardrails, PII detection, or content filtering is preferred.
- Experience with GCP alongside AWS is preferred.
Benefits
- Group Medical, Personal Accident, and Term Life Insurance
- Dental, vision, and OPD coverage
- Mental health support
- Fitness benefit
- Car lease policy
- Gratuity
Tech Stack
Categories
DevOpsSite Reliability
About Demandbase
Demandbase builds a B2B account-based go-to-market platform that unifies account intelligence, intent data, programmatic advertising, and sales intelligence for marketing and sales teams. Delivered as SaaS, it helps companies identify, target, and engage buying groups and measure pipeline impact across the customer lifecycle. Founded in 2005 and headquartered in San Francisco, the company is privately held and serves enterprises pursuing account-based marketing and revenue operations.