2 months ago
Pune, IndiaStaff+
Responsibilities
- Develop and implement comprehensive test plans and strategies for software and GenAI applications.
- Create and execute automated test scripts and scalable test automation solutions using Python and testing frameworks.
- Test and validate LLMs, generative AI, agentic AI, machine learning models, NLU, and NLP quality.
- Perform UI, mobile, API, performance, load, and regression testing as applicable.
- Establish quality gates, governance frameworks, and testing excellence practices for GenAI applications.
- Analyze requirements, participate in design discussions, define acceptance criteria, and conduct root-cause analysis of defects.
- Collaborate with development, product, operations, and other cross-functional teams to resolve defects and improve quality.
- Design and optimize CI/CD pipelines and enterprise-scale cloud-based testing solutions.
- Participate in code reviews, promote code quality and knowledge sharing, and contribute to technology communities.
- Lead quality initiatives, manage risk and controls, and guide colleagues through complex technical assignments.
Requirements
- Expert proficiency in Large Language Models, LangChain, and LangGraph.
- Advanced Python programming skills with experience architecting scalable test automation solutions.
- Hands-on production experience with generative AI and agentic AI implementations.
- Strong knowledge of generative AI architecture, evaluation methodologies, machine learning model testing, NLU testing, and NLP quality assurance.
- Comprehensive knowledge of AWS services, particularly Bedrock, Lambda, and CloudWatch.
- Experience architecting and implementing enterprise-scale cloud-based testing solutions.
- Expert proficiency with Git, Stash, and branching strategies.
- Experience designing and optimizing CI/CD pipelines using Jenkins, GitLab CI, or similar tools.
- Deep understanding of Agile/Scrum, SDLC best practices, and DevOps culture.
- Experience leading cross-functional collaboration and establishing quality gates and governance frameworks.
- Expert-level knowledge of UI, mobile, and API testing methodologies.
- Advanced proficiency with BDD frameworks and test design patterns.
- Experience with Jira, Xray, ALM, and TestRail.
- Preferred experience with prompt engineering, LLM evaluation frameworks such as RAGAS and DeepEval, vector databases for RAG testing, GenAI performance and load testing, AI ethics and bias detection, AWS or Azure AI certifications, Docker, Kubernetes, data engineering, or data quality testing.
- Ability to work with risk and controls, change and transformation, business acumen, strategic thinking, and digital and technology.
Benefits
- The role is based in Pune.
- The position is an Assistant Vice President role with opportunities to lead quality initiatives and guide team members.
- The role offers cross-functional exposure across development, product, operations, risk, controls, and technology governance.
About Barclays
Our vision is to be the UK-centred leader in global finance. We are a diversified bank with comprehensive UK consumer, corporate and wealth and private banking franchises, a leading investment bank and a strong, specialist US consumer bank. Through these five divisions, we are working together for a better financial future for our customers, clients, and communities. For further information about Barclays, please visit home.barclays.