5 hours ago
London, United KingdomMid Level
Responsibilities
- Develop and execute end-to-end automated tests across GenAI evaluations, APIs, UIs, and performance scenarios using Python.
- Implement testing strategies for conversational AI, agentic systems, and LLM-powered workflows.
- Maintain automated prompt-testing and regression suites to identify unintended changes in model behavior.
- Support evaluation frameworks measuring accuracy, relevance, safety, consistency, and latency.
- Curate test datasets and golden examples covering diverse user scenarios and edge cases.
- Assist with adversarial testing for hallucinations and biases and support production alerting integration.
- Help developers validate GenAI code, support user acceptance testing, and document test procedures and known issues.
- Own automated coverage for the assigned workstream.
Requirements
- QA or test engineering experience with exposure to or interest in AI/ML systems.
- Hands-on Python coding experience.
- Understanding of GenAI technologies, LLMs, prompt engineering, and AI application patterns.
- Experience with test automation frameworks and scripting, including Python, Pytest, API testing, and UI testing.
- Knowledge of functional, integration, regression, performance, and security testing.
- Ability to execute test cases and evaluation criteria for non-deterministic systems.
- Experience with API testing tools such as Postman and REST Assured and with backend testing.
- Familiarity with CI/CD pipelines and automated testing integration.
About Anaplan
Anaplan builds a cloud platform for enterprise planning, forecasting, and scenario modeling used by finance, supply chain, sales, and workforce teams. It sells subscription software and related services to large organizations that need connected, cross-functional planning. Founded in 2006 and headquartered in Miami, Florida, Anaplan was acquired by Thoma Bravo in 2022 after previously trading on the NYSE.