24 hours ago
Pune, India or Chennai, IndiaSenior
Responsibilities
- Design and execute comprehensive test strategies, plans, and test cases for AI/ML and Generative AI applications.
- Build and maintain scalable test automation frameworks using Python and Selenium.
- Perform data validation and integrity testing with advanced SQL queries.
- Evaluate AI model performance, accuracy, fairness, robustness, and suitability against business and technical requirements.
- Design evaluation frameworks for Generative AI, LLM, and RAG applications using metrics such as accuracy, relevance, precision, faithfulness, context recall, and response quality.
- Implement AI observability and monitoring solutions and automated quality evaluation frameworks.
- Establish AI guardrails and governance controls for bias, toxicity, hallucinations, prompt injection, jailbreak attempts, and unsafe outputs.
- Identify, document, and track defects and performance issues through resolution.
- Lead UAT, product rollout, quality monitoring, audits, standards compliance, corrective actions, traceability, quality metrics, and root-cause analysis.
- Mentor junior team members and improve quality engineering standards, procedures, and methodologies.
Requirements
- 7+ years of professional experience in Quality Engineering or Software/Data Quality Assurance.
- Strong Python proficiency for test automation and framework development.
- Hands-on experience with Selenium or similar web automation tools.
- Strong SQL expertise for complex queries and backend data validation.
- Experience testing AI and machine learning systems; Generative AI familiarity is highly desirable.
- Deep understanding of quality engineering fundamentals, testing methodologies, and the software development lifecycle.
- Experience evaluating Generative AI, LLM, and RAG applications and benchmarking models such as GPT and Gemini.
- Experience with AI observability platforms such as Langfuse, Arize AI, and Fiddler AI.
- Experience with AI quality evaluation frameworks such as RAGAS, Promptfoo, DeepEval, and TruLens.
- Excellent analytical, problem-solving, communication, collaboration, and attention-to-detail skills.
- Bachelor’s or university degree, or equivalent experience.
About Citi
Citi is a public financial-services company offering consumer and institutional banking, credit cards, wealth management, treasury and trade solutions, and capital-markets services. It serves individuals, corporations, financial institutions, and governments in more than 160 countries and jurisdictions, earning interest and fee income from lending, payments, trading, and advisory. Founded in 1812 and headquartered in New York, it trades on the NYSE under the ticker C.