17 hours ago
Remote, United StatesMid Level
H1B sponsor
Responsibilities
- Develop and execute end-to-end automated test coverage for GenAI evaluations, APIs, UIs, and performance tests using Python.
- Create testing strategies for conversational AI, agentic systems, and LLM-powered workflows.
- Maintain automated prompt-testing and regression-test suites to detect unintended changes in model behavior.
- Support evaluation frameworks measuring accuracy, relevance, safety, consistency, and latency.
- Curate test datasets and golden examples representing diverse user scenarios and edge cases.
- Assist with adversarial testing for hallucinations and bias and support production alerting integration.
- Maintain internal testing frameworks, support developer validation and user acceptance testing, and document procedures and known issues.
Requirements
- Solid QA or test engineering experience with exposure to or interest in AI/ML systems.
- Hands-on Python coding experience.
- Understanding of GenAI technologies, LLMs, prompt engineering, and AI application patterns.
- Experience with test automation frameworks and scripting, including Python, Pytest, API testing, and UI testing.
- Knowledge of functional, integration, regression, performance, and security testing.
- Ability to evaluate test criteria for non-deterministic systems.
- Experience with API testing tools such as Postman and REST Assured and with backend testing.
- Familiarity with CI/CD pipelines and automated testing integration.
About Anaplan
Anaplan builds a cloud platform for enterprise planning, forecasting, and scenario modeling used by finance, supply chain, sales, and workforce teams. It sells subscription software and related services to large organizations that need connected, cross-functional planning. Founded in 2006 and headquartered in Miami, Florida, Anaplan was acquired by Thoma Bravo in 2022 after previously trading on the NYSE.