Lead AI Ops Engineer
Mastercard21 hours ago
Gurgaon, IndiaStaff+
Responsibilities
- Own production monitoring for one or more AI agents, including metrics, KPIs, accuracy, latency, SLA breaches, guardrail triggers, and out-of-scope rates.
- Provide L1/L2 production support, including first-response monitoring, triage, and escalation for agentic services.
- Diagnose failures using eval scores, drift alerts, traces, and error patterns across intent, tools, parameters, and hallucinations.
- Evaluate production reasoning, context retrieval, resolutions against ground truth, and learning-loop drift.
- Manage post-production reliability issues involving APIs, authentication, and external AI endpoints.
- Partner with AI Engineering to test prompts, context, and routing changes and validate improvements through evaluations.
- Embed governance, approval workflows, and audit logging into the agent learning loop.
Requirements
- Experience monitoring or operating production AI/ML or agentic systems, including observability, evaluation, and drift detection.
- Strong analytical ability to diagnose intent classification, tool selection, parameter, and hallucination issues.
- Understanding of AI/LLM observability, tracing, telemetry, guardrails, benchmarks, automated evaluations, human review, and regression testing.
- Familiarity with governance, compliance, responsible AI, PII detection, access control, audit logging, and policy enforcement is preferred.
- Ability to work independently with technology teams while owning an agent's post-launch health.
- Strong communication skills for translating production signals into prioritized requests for AI Product and AI Engineering teams.
- Comfort operating and improving processes in a fast-evolving environment.
Categories
Site Reliability
About Mastercard
Mastercard builds and operates a global payments network used by banks, merchants, fintechs, and governments, offering card processing, real-time payments, tokenization, and fraud/risk services. It generates revenue from transaction processing and assessment/service fees across more than 200 countries and territories. Founded in 1966 and headquartered in Purchase, New York, Mastercard is a public company listed on the NYSE (ticker: MA).