5 hours ago
London, United KingdomSenior
Responsibilities
- Design and architect test plans, evaluation frameworks, and quality metrics for LLM-powered applications and agentic workflows.
- Develop automated end-to-end test suites for evaluations, API checks, UI checks, and performance tests.
- Create automated prompt-testing systems and regression suites to detect unintended model-behavior changes.
- Define multidimensional evaluation criteria covering accuracy, relevance, safety, consistency, and latency.
- Perform adversarial red-team testing to identify hallucinations, bias, and security vulnerabilities.
- Develop internal testing frameworks that help software engineering teams validate GenAI implementations.
- Deploy real-time monitoring and alerting to detect quality degradation in production AI features.
- Own quality and testing strategy, maintain golden-example datasets, and define QA best practices.
Requirements
- Deep understanding of GenAI technologies, LLMs, prompt engineering, and AI application patterns.
- Strong hands-on Python coding experience.
- Expertise with test automation frameworks and scripting using Python, JavaScript, Selenium, and Pytest.
- Strong background in functional, integration, regression, performance, and security testing methodologies.
- Ability to design test cases and evaluation criteria for non-deterministic systems.
- Experience with API testing tools including Postman and REST Assured, plus backend testing.
- Strong experience integrating automated testing with CI/CD pipelines.
- Strong analytical, problem-solving, and attention-to-detail skills.
Tech Stack
Categories
About Anaplan
Anaplan builds a cloud platform for enterprise planning, forecasting, and scenario modeling used by finance, supply chain, sales, and workforce teams. It sells subscription software and related services to large organizations that need connected, cross-functional planning. Founded in 2006 and headquartered in Miami, Florida, Anaplan was acquired by Thoma Bravo in 2022 after previously trading on the NYSE.