8 hours ago
Montréal, CanadaSenior
Responsibilities
- Design and execute comprehensive test strategies for AI systems, LLMs, RAG pipelines, knowledge bases, and securitization platforms.
- Evaluate prompts and generated outputs for accuracy, tone, coherence, hallucinations, bias, safety, and off-target behavior.
- Test vector database retrieval, similarity thresholds, embedding drift, edge cases, and performance at scale.
- Use LangChain and LangGraph to analyze chains and graphs, identify failure points, and create test harnesses.
- Validate MCP integration points, tool availability, and error handling.
- Design, build, and maintain automated test suites for functional, regression, integration, performance, API, and backend testing.
- Validate end-to-end securitization workflows, data integrity, upstream and downstream systems, reconciliations, and reporting.
- Coordinate regression testing for releases, patches, and infrastructure changes while ensuring stability and backward compatibility.
- Establish QA standards covering traditional quality assurance and AI-specific validation.
Requirements
- More than 7 years of experience in quality assurance or quality engineering.
- Strong knowledge of securitization, capital markets, or similar asset classes.
- Hands-on experience with test automation tools such as Selenium, Robot Framework, and Playwright.
- Proficiency in Java and Python, with demonstrated experience implementing automation frameworks.
- Experience with API test automation and backend system validation.
- Proficiency in database query development, data validation, and reconciliation testing.
- Experience with CI/CD pipelines and DevOps practices, including Jenkins and GitHub or comparable tools.
- Core understanding of LLM architecture and behavior, including tokenization, embeddings, attention mechanisms, and inference.
- Hands-on experience with LangChain and/or LangGraph, plus knowledge of RAG pipelines, vector databases, and agentic solutions.
- Familiarity with Model Context Protocols and integration testing.
- Understanding of AI bias, safety, and red-team testing methodologies.
Benefits
- Hybrid work arrangement requiring four days per week at a Cognizant or client office in Montreal, Quebec.
- Work-life balance and wellbeing programs.
- Work arrangements may change according to project, business, and client requirements.
About Cognizant
Cognizant is a public IT services and consulting firm that designs, builds, and runs enterprise technology, including digital engineering, cloud modernization, data/AI, and managed services. It sells consulting, systems integration, and outsourcing on multi-year engagements to large enterprises in healthcare, banking, retail, communications, and manufacturing. Founded in 1994 and headquartered in Teaneck, New Jersey, Cognizant is NASDAQ-listed (CTSH) and a Fortune 500 company.