5 hours ago
London, United KingdomSenior
Responsibilities
- Design comprehensive test plans, evaluation frameworks, and quality metrics for LLM-powered applications and agentic workflows.
- Develop end-to-end automated test suites in Python for evaluations, API checks, UI checks, and performance tests.
- Create automated prompt-testing systems and regression suites to detect unintended changes in model behavior.
- Measure GenAI quality across accuracy, relevance, safety, consistency, and latency.
- Perform adversarial testing to identify hallucinations, bias, and security vulnerabilities.
- Build internal testing tools and frameworks that help software engineering teams validate GenAI implementations.
- Deploy real-time monitoring and alerting systems for production AI quality degradation.
- Own quality strategy, maintain golden-example test datasets, and define QA best practices.
Requirements
- Deep understanding of GenAI technologies, LLMs, prompt engineering, and AI application patterns.
- Strong hands-on Python coding experience.
- Expert experience with test automation frameworks and scripting using Python, JavaScript, Selenium, and Pytest.
- Strong background in functional, integration, regression, performance, and security testing.
- Ability to design test cases and evaluation criteria for nondeterministic systems.
- Experience with API testing tools including Postman and REST Assured, plus backend testing experience.
- Strong experience integrating automated testing with CI/CD pipelines.
- Strong analytical, problem-solving, and attention-to-detail skills.
Tech Stack
Categories
About Anaplan
Anaplan builds a cloud platform for enterprise planning, forecasting, and scenario modeling used by finance, supply chain, sales, and workforce teams. It sells subscription software and related services to large organizations that need connected, cross-functional planning. Founded in 2006 and headquartered in Miami, Florida, Anaplan was acquired by Thoma Bravo in 2022 after previously trading on the NYSE.