22 hours ago
Dublin, IrelandSenior
Responsibilities
- Design and build AI-augmented automation systems and scalable frameworks for web UI, APIs, and data pipelines.
- Use Playwright, TypeScript, and AI coding agents such as Claude Code to create, refactor, and maintain test suites.
- Translate acceptance criteria into maintainable tests and define reusable patterns that enable engineers to own automation.
- Develop risk-based testing strategies across unit, integration, contract, and end-to-end layers, including quality gates and release-readiness standards.
- Create evaluation pipelines for LLM-powered features using golden datasets, regression harnesses, semantic similarity scoring, and LLM-as-a-judge patterns.
- Test agentic workflows and identify prompt drift, hallucinations, retrieval failures, schema regressions, prompt injection, jailbreaks, unsafe tool use, and data leakage.
- Integrate automation and evaluation into CI/CD pipelines and support releases with reliable signals and rollback readiness.
- Investigate distributed-system issues using logs, traces, and metrics, improve LLM trace visibility, and lead defect triage and reporting.
- Use quality metrics such as leakage, flake rate, MTTR, and escape rate to drive continuous improvement.
- Embed security, performance, resilience, scalability, and AI-specific risk testing into quality practices.
- Partner with software engineers, data scientists, product managers, and SRE to embed quality throughout the development lifecycle.
Requirements
- Proven experience as a Senior QA Automation Engineer or SDET in production environments.
- Strong hands-on experience with Playwright or an equivalent tool using TypeScript.
- Demonstrated daily use of AI coding tools such as Claude Code.
- Strong software engineering fundamentals and ability to write maintainable, production-grade code.
- Experience designing automation frameworks and end-to-end test strategies.
- Deep understanding of CI/CD and integrating automation into delivery pipelines.
- Strong debugging and observability skills across distributed systems.
- Experience driving shift-left practices and enabling engineers to own quality.
- Understanding of databases, APIs, and test-data validation using SQL, REST, or GraphQL.
- Security-aware testing experience, including OWASP and secure test design.
- Experience with AI/LLM system testing, evaluation frameworks, prompt regression, drift detection, LLM-as-a-judge evaluation, semantic scoring, or golden datasets is particularly valuable.
- Exposure to RAG pipelines, vector search, knowledge graph systems, LLM observability and experimentation tools such as Langfuse or MLflow, and agentic multi-step workflows is particularly valuable.
- Performance and load testing experience at scale, cloud experience with AWS or GCP, and test infrastructure design experience are particularly valuable.
- Experience in high-trust domains such as media, intelligence, risk, or compliance is particularly valuable.
Benefits
- Benefits-eligible employees receive a comprehensive benefits package covering health, retirement, wellbeing, and optional benefits.
- The employer offers competitive and flexible compensation based on experience, skills, location, and other relevant factors; no specific base-pay amount is stated.
Tech Stack
Categories
About Dow Jones
Dow Jones provides news, market data, and risk/compliance information for consumers and enterprises through brands and platforms such as The Wall Street Journal, Barron’s, MarketWatch, Factiva, OPIS, and Dow Jones Risk & Compliance. Its business model spans consumer subscriptions, enterprise licensing, advertising, and data services used by financial institutions, corporations, and investors. Founded in 1882 and headquartered in New York, it is a subsidiary of News Corp.