24 hours ago
Pune, India or Chennai, IndiaSenior
Responsibilities
- Design and execute comprehensive test strategies, plans, and test cases for AI/ML and Generative AI applications.
- Build and maintain scalable Python and Selenium test automation frameworks.
- Perform SQL-based data validation and integrity testing.
- Evaluate AI model accuracy, fairness, robustness, relevance, faithfulness, context recall, and response quality.
- Design evaluation frameworks for Generative AI, LLM, and RAG applications and benchmark models against business and technical requirements.
- Implement AI observability and monitoring using Langfuse, Arize AI, and Fiddler AI.
- Implement automated AI quality evaluation with RAGAS, Promptfoo, DeepEval, and TruLens.
- Establish guardrails and governance controls for bias, toxicity, hallucinations, prompt injection, jailbreak attempts, and unsafe outputs.
- Collaborate with data scientists, software engineers, and product managers to integrate testing into the development lifecycle.
- Identify, document, track, and resolve defects and performance issues.
- Lead UAT, product rollout, process monitoring, audits, quality metrics, root-cause analysis, corrective actions, and standards compliance.
- Mentor junior team members and champion quality engineering process improvements.
Requirements
- 7+ years of professional experience in Quality Engineering or Software/Data Quality Assurance.
- Strong proficiency in Python scripting for test automation and framework development.
- Hands-on experience with Selenium or comparable web automation tools.
- Solid SQL expertise, including complex queries and backend data validation.
- Experience testing AI and machine learning systems; familiarity with Generative AI is highly desirable.
- Deep understanding of quality engineering fundamentals, testing methodologies, and the software development lifecycle.
- Experience evaluating Generative AI, LLM, and RAG applications and assessing foundation models such as GPT and Gemini.
- Strong analytical, problem-solving, communication, collaboration, and attention-to-detail skills.
- Bachelor’s or university degree, or equivalent experience.
About Citi
Citi is a public financial-services company offering consumer and institutional banking, credit cards, wealth management, treasury and trade solutions, and capital-markets services. It serves individuals, corporations, financial institutions, and governments in more than 160 countries and jurisdictions, earning interest and fee income from lending, payments, trading, and advisory. Founded in 1812 and headquartered in New York, it trades on the NYSE under the ticker C.