1 day ago
Cluj-Napoca, RomaniaStaff+
Responsibilities
- Lead Quality Engineering across multiple AI delivery pods and establish consistent, risk-based quality standards.
- Define test strategies, acceptance criteria, UAT approaches, quality risks, quality gates, and release-readiness criteria with Product and Engineering.
- Ensure coverage across functional, integration, end-to-end, non-functional, AI-specific, and business acceptance testing.
- Define evaluation approaches for LLM, RAG, agentic, workflow-based, and AI-enabled solutions, including grounding, hallucination risk, regression, tool calling, workflow behavior, and non-deterministic outputs.
- Establish AI evaluation datasets, test harnesses, baselines, thresholds, and regression approaches.
- Guide and implement reusable automation across web applications, APIs, backend services, integrations, asynchronous workflows, and AI components.
- Ensure UAT readiness through appropriate evidence, environments, test data, defect status, entry and exit criteria, and known-risk tracking.
- Report quality metrics, risks, defect trends, AI evaluation results, UAT readiness, and release readiness to pod and program leadership.
- Conduct quality reviews across pods and improve testing, testability, observability, and quality evidence.
- Use production incidents, escaped defects, telemetry, and user feedback to improve quality and evaluation strategies.
- Coach engineers and delivery teams on shared ownership of quality across the software lifecycle.
Requirements
- Proven experience leading Quality Engineering for enterprise-scale AI or GenAI solutions in production.
- Hands-on experience testing LLM, RAG, agentic, workflow-based, or AI-enabled applications.
- Strong experience defining end-to-end test strategies, UAT approaches, quality gates, and release-readiness criteria.
- Experience working across multiple delivery teams or pods in complex enterprise environments.
- Strong software engineering and automation skills in TypeScript/JavaScript and/or Python.
- Experience with automation frameworks such as Playwright, Cypress, Selenium, Jest, Mocha, or PyTest.
- Strong experience testing web applications, APIs, backend services, integrations, and distributed systems.
- Experience with AI evaluation frameworks, regression suites, test harnesses, and representative evaluation datasets.
- Understanding of CI/CD, cloud-native systems, observability, and Secure SDLC practices.
- Strong stakeholder management skills and ability to translate quality evidence into risks, metrics, and leadership recommendations.
- Preferred experience with LangChain, LangSmith, Mastra, or similar AI engineering and evaluation tooling.
- Preferred experience testing complex RAG, multi-agent, or tool-calling solutions.
- Preferred experience with performance, resilience, accessibility, or security testing and with using production telemetry and incidents to improve testing.
Benefits
- Professional development opportunities, interesting work, supportive leaders, and an inclusive culture.
- Yearly budget and flexible benefits package of up to 20 percent of annual salary.
- 30+ days off, including legal days, birthday, public holiday replacements, and benefits options.
- Performance bonus scheme.
- Matching charity contributions, charity days off, and the Pay it Forward charity challenge.
- Pension, life and medical insurance, meal vouchers, and travel insurance.
- Hybrid work arrangement requiring at least three days per week in the local office or onsite with clients; office-based teams identify at least one weekly anchor day.
Tech Stack
AWSAzureCypressFastAPIGitHub ActionsGitLab CI/CDJavaScriptJestMochaNestJSNode.jspytestPythonReactSeleniumTypeScript
Categories
About Marsh
Marsh is a global insurance broker and risk advisor that helps businesses, public entities, and individuals assess, transfer, and manage risk through insurance placement, captives, and advisory services. It sells brokerage, risk consulting, and claims management across industries, including cyber and property, and operates in more than 130 countries. Marsh is a business of Marsh McLennan and is headquartered in New York.