18 hours ago
Pune, IndiaStaff+
Responsibilities
- Design and execute end-to-end tests for single-agent, multi-agent, Agentic AI, and RAG workflows.
- Validate agent reasoning, planning, tool use, memory, handoffs, termination behavior, and loop prevention.
- Test RAG ingestion, chunking, embeddings, retrieval, reranking, generation, context management, citations, and conversational coherence.
- Build automated test harnesses, trajectory replay systems, tool mocks, prompt simulation, evaluation pipelines, and prompt regression suites.
- Create data validation and quality frameworks for training, retrieval, and inference data.
- Conduct AI safety, red-team, adversarial, privacy, bias, fairness, security, and privilege-control testing.
- Test AI APIs, inference endpoints, orchestration services, vector databases, auto-scaling, failure recovery, and cost efficiency under load.
- Establish AI system SLOs and SLAs and collaborate with MLOps teams on observability, monitoring, alerting, anomaly detection, and chaos engineering.
- Document red-team findings, severity ratings, remediation recommendations, and safety testing playbooks.
Requirements
- Bachelor’s degree in Computer Science, Engineering, or a related field is required; a master’s degree is a plus.
- At least 8 years of experience in software or AI/ML quality engineering is required.
- At least 3 years of hands-on experience with RAG systems or Agentic AI is required.
- Proven experience building automated test frameworks for non-deterministic AI systems.
- Strong background in performance testing and AI safety/security assessments.
- Deep understanding of RAG architectures, agent design patterns, AI safety and alignment, token economics, context management, and distributed AI performance engineering.
- Preferred experience with MCP or similar agentic communication standards, multimodal agent testing, and regulated industries such as banking, finance, or healthcare.
- Preferred familiarity with Chaos Monkey, Gremlin, or LitmusChaos.
Benefits
- Full-time position.
- Citi is an equal opportunity employer and provides reasonable accommodations for applicants with disabilities.
Categories
About Citi
Citi is a public financial-services company offering consumer and institutional banking, credit cards, wealth management, treasury and trade solutions, and capital-markets services. It serves individuals, corporations, financial institutions, and governments in more than 160 countries and jurisdictions, earning interest and fee income from lending, payments, trading, and advisory. Founded in 1812 and headquartered in New York, it trades on the NYSE under the ticker C.