2 days ago
London, United KingdomMid Level
Responsibilities
- Develop and execute end-to-end automated tests across GenAI evaluations, APIs, UIs, and performance scenarios using Python.
- Create testing strategies for conversational AI, agentic systems, and LLM-powered workflows.
- Maintain prompt-testing suites and regression tests to detect unintended model behavior changes.
- Support evaluation frameworks measuring accuracy, relevance, safety, consistency, and latency.
- Curate test datasets and golden examples covering diverse user scenarios and edge cases.
- Assist with adversarial testing for hallucinations and bias, production alerting integration, developer validation tools, and user acceptance testing.
- Own automated coverage for the assigned workstream and document testing procedures and known issues.
Requirements
- Solid QA or test engineering experience, with exposure to or interest in AI/ML systems.
- Hands-on Python coding experience.
- Understanding of GenAI technologies, LLMs, prompt engineering, and AI application patterns.
- Experience with test automation frameworks and scripting, including Python, Pytest, API testing, and UI testing.
- Knowledge of functional, integration, regression, performance, and security testing methodologies.
- Experience with API testing tools such as Postman and REST Assured, plus backend testing.
- Familiarity with CI/CD pipelines and automated testing integration.
About Anaplan
Anaplan builds a cloud platform for enterprise planning, forecasting, and scenario modeling used by finance, supply chain, sales, and workforce teams. It sells subscription software and related services to large organizations that need connected, cross-functional planning. Founded in 2006 and headquartered in Miami, Florida, Anaplan was acquired by Thoma Bravo in 2022 after previously trading on the NYSE.