17 hours ago
Responsibilities
- Design and architect test plans, evaluation frameworks, and quality metrics for LLM-powered applications and agentic workflows.
- Develop end-to-end automated test suites in Python for evaluations, API checks, UI checks, and performance tests.
- Create automated prompt-testing systems and regression suites to detect unintended changes in model behavior.
- Define evaluation criteria for accuracy, relevance, safety, consistency, and latency.
- Perform adversarial testing to identify hallucinations, bias, and security vulnerabilities.
- Develop internal testing tools and frameworks for engineering teams validating GenAI implementations.
- Deploy real-time monitoring and alerting systems for AI feature quality degradation in production.
- Own the quality and testing strategy for the workstream, including golden-example datasets and QA best practices.
Requirements
- Deep understanding of GenAI technologies, LLMs, prompt engineering, and AI application patterns.
- Strong hands-on Python coding experience.
- Expert experience with test automation frameworks and scripting, including Python, JavaScript, Selenium, and Pytest.
- Strong background in functional, integration, regression, performance, and security testing.
- Ability to design test cases and evaluation criteria for non-deterministic systems.
- Experience with API testing tools including Postman and REST Assured, as well as backend testing.
- Strong experience integrating automated testing with CI/CD pipelines.
- Strong analytical and problem-solving skills with attention to detail.
Tech Stack
Categories
About Anaplan
Anaplan builds a cloud platform for enterprise planning, forecasting, and scenario modeling used by finance, supply chain, sales, and workforce teams. It sells subscription software and related services to large organizations that need connected, cross-functional planning. Founded in 2006 and headquartered in Miami, Florida, Anaplan was acquired by Thoma Bravo in 2022 after previously trading on the NYSE.