1 day ago
Pune, IndiaStaff+
Responsibilities
- Design and execute end-to-end tests for single-agent, multi-agent, and RAG workflows, including reasoning, tool use, memory, handoffs, retrieval, reranking, generation, and citations.
- Build automated test harnesses, agent trajectory replay, tool mocks, prompt simulations, regression suites, and continuous AI evaluation pipelines.
- Create data-quality and vector database validation frameworks for training, retrieval, and inference data.
- Conduct AI safety and security testing, including red-teaming, jailbreak and prompt-injection testing, guardrail validation, privacy and PII leakage testing, bias assessment, and privilege-control verification.
- Perform load, stress, soak, spike, performance, cost, reliability, and chaos testing for AI APIs, inference endpoints, agent orchestration services, and vector databases.
- Establish AI-system SLOs and SLAs and collaborate with MLOps teams on observability dashboards, monitoring alerts, and anomaly detection.
- Document safety findings, severity ratings, remediation recommendations, and testing playbooks.
- Collaborate with cybersecurity and MLOps teams on threat modeling, vulnerability management, resilience, and production monitoring.
Requirements
- Bachelor’s degree in Computer Science, Engineering, or a related field.
- 8+ years of experience in software or AI/ML quality engineering.
- 3+ years of hands-on experience with RAG systems or Agentic AI.
- Proven experience building automated test frameworks for non-deterministic AI systems.
- Strong background in performance testing and AI safety/security assessments.
- Deep understanding of RAG architectures, agent design patterns, AI safety and alignment, token economics, context management, and distributed AI performance engineering.
- Preferred experience with MCP or similar agentic communication standards and multi-modal agent testing.
- Preferred experience in banking, finance, healthcare, or other regulated industries.
- Preferred familiarity with Chaos Monkey, Gremlin, or LitmusChaos.
About Citi
Citi is a public financial-services company offering consumer and institutional banking, credit cards, wealth management, treasury and trade solutions, and capital-markets services. It serves individuals, corporations, financial institutions, and governments in more than 160 countries and jurisdictions, earning interest and fee income from lending, payments, trading, and advisory. Founded in 1812 and headquartered in New York, it trades on the NYSE under the ticker C.