AI Automation QA (Night Shift)
Willis Towers Watson1 day ago
Taguig City, PhilippinesMid Level
Responsibilities
- Define quality criteria and testing strategies for AI agent workflows.
- Build automated evaluation harnesses for hallucination rates, tool misuse, policy violations, and task success.
- Develop and maintain automated UI, API, and workflow test suites.
- Continuously monitor production systems for anomalies, quality degradation, and safety concerns.
- Apply LLM evaluation frameworks to analyze output quality, regression, and system drift.
- Create dashboards and reports communicating quality metrics and trends.
- Maintain runbooks and contribute to incident response activities.
- Collaborate with developers to improve prompts, tool definitions, and workflow designs.
- Ensure testing, logging, and monitoring align with privacy, audit, and regulatory requirements.
Requirements
- At least three years of experience in QA, test automation, or DevOps, or two years of direct experience testing AI or ML-enabled systems.
- Strong Python skills for test automation, evaluation harnesses, and basic data analysis.
- Experience with Playwright for end-to-end workflow testing.
- Experience with Deepeval, RAGAS, and Evidently.AI for LLM quality, drift, and regression analysis.
- Experience validating APIs and agent workflows with custom scripts.
- Pytest or equivalent testing framework experience is desirable.
- SQL experience for querying logs, metrics, or evaluation datasets is desirable.
- Prometheus, Grafana, or similar monitoring-tool experience is desirable.
- Familiarity with hallucination detection, AI safety patterns, CI/CD pipelines, and Git-based workflows is desirable.
- Strong attention to detail and ability to use evidence-based insights in collaborative product and engineering decisions.
Benefits
- Night-shift work schedule.
- Equal opportunity employer.
About Willis Towers Watson
WTW is a global advisory, broking, and software firm serving employers, insurers, and institutional clients across people, risk, and capital. It provides insurance brokerage and risk advisory, employee benefits and pensions consulting, and HR/benefits administration software and analytics. Headquartered in London and listed on NASDAQ (WTW), the company was formed in 2016 via the Willis Group –Towers Watson merger and operates in roughly 140 markets worldwide.