5 hours ago
London, United KingdomAny Level
Responsibilities
- Develop and execute automated end-to-end tests across evaluations, APIs, UIs, and performance scenarios using Python.
- Create testing strategies for conversational AI, agentic systems, and LLM-powered workflows.
- Maintain prompt-testing suites and regression tests that detect unintended changes in model behavior.
- Support evaluation frameworks measuring accuracy, relevance, safety, consistency, and latency.
- Curate test datasets and golden examples representing diverse scenarios and edge cases.
- Conduct adversarial testing for hallucinations and bias and support production alerting integration.
- Maintain automated coverage, document testing procedures and known issues, and assist with user acceptance testing.
Requirements
- Solid QA or test engineering experience with exposure to or interest in AI/ML systems.
- Hands-on Python coding experience.
- Understanding of GenAI technologies, LLMs, prompt engineering, and AI application patterns.
- Experience with test automation frameworks and scripting, including Pytest, API testing, and UI testing.
- Knowledge of functional, integration, regression, performance, and security testing methodologies.
- Ability to test and evaluate non-deterministic systems.
- Experience with API testing tools such as Postman and REST Assured and with backend testing.
- Familiarity with CI/CD pipelines and automated testing integration.
About Anaplan
Anaplan builds a cloud platform for enterprise planning, forecasting, and scenario modeling used by finance, supply chain, sales, and workforce teams. It sells subscription software and related services to large organizations that need connected, cross-functional planning. Founded in 2006 and headquartered in Miami, Florida, Anaplan was acquired by Thoma Bravo in 2022 after previously trading on the NYSE.