Senior Product Software Engineer - AI Quality Engineering
Wolters Kluwer1 day ago
Base Salary
$83k - $146k/yr
Responsibilities
- Own the quality engineering strategy and implementation for enterprise-scale generative AI applications, including RAG pipelines, LLM orchestration, agentic workflows, APIs, ingestion pipelines, and AI-driven user experiences.
- Architect evaluation harnesses measuring retrieval accuracy, citation correctness, relevance, groundedness, hallucination rate, safety, latency, cost, and end-to-end behavior.
- Design automated regression suites covering prompts, embeddings, chunking strategies, model versions, retrieval configuration, orchestration logic, and application workflows.
- Validate AI architecture, service integrations, API behavior, data ingestion, observability, resilience, security controls, and production readiness with development teams.
- Lead root-cause analysis for AI quality failures using logs, traces, evaluation outputs, feedback signals, customer scenarios, and domain-specific acceptance criteria.
- Define and maintain measurable AI release-quality gates and advise leaders on risk, readiness, and tradeoffs.
- Develop Python test automation for APIs, services, chat workflows, data pipelines, and AI evaluation workflows within delivery pipelines.
- Build observability practices and dashboards tracking model behavior, retrieval performance, operational health, quality trends, and customer-impacting failures.
- Implement responsible AI validation covering bias, fairness, content safety, prompt injection, data privacy, and compliance evidence.
- Translate customer workflows and business requirements into testable AI quality criteria and mentor engineers and quality team members in AI evaluation practices.
Requirements
- At least 5 years of experience in quality engineering, SDET, software engineering, test automation, reliability engineering, or equivalent engineering roles.
- At least 2 years of hands-on experience testing, evaluating, or validating AI/ML, LLM, RAG, or other non-deterministic systems in production or production-like environments.
- Strong Python test automation experience with frameworks such as Pytest or unittest, including reusable test harnesses and regression suites.
- Experience validating APIs, microservices, service integrations, data ingestion pipelines, and end-to-end web or chat workflows.
- Practical understanding of RAG systems, vector search, embeddings, prompt behavior, model evaluation, groundedness, citation quality, and AI failure modes.
- Experience defining quality metrics and success criteria for probabilistic outputs.
- Knowledge of delivery pipelines, Git-based workflows, automated regression testing, release gates, and modern engineering practices.
- Experience with observability, logging, tracing, dashboards, and production metrics.
- Ability to decompose complex AI behaviors into testable risks and communicate findings to technical and non-technical stakeholders.
- Experience collaborating across product, design, development, and domain expert teams in Agile environments.
- Commitment to responsible AI practices involving trust, transparency, security, privacy, fairness, and customer impact.
- Preferred experience with LLM observability and evaluation platforms such as LangSmith, Langfuse, TruLens, Arize/Phoenix, or custom tooling.
- Preferred experience testing RAG pipelines with Azure AI Search, Pinecone, Weaviate, Elasticsearch, or similar technologies.
- Preferred experience with LangChain, LlamaIndex, Semantic Kernel, Azure OpenAI Service, Azure AI Studio, Azure cloud services, Docker, Kubernetes, Locust, k6, JMeter, Azure Application Insights, Grafana, Prometheus, Datadog, or OpenTelemetry.
- Preferred knowledge of LLMOps, model monitoring, A/B testing, human-in-the-loop evaluation, production feedback loops, and sensitive or regulated data.
- Background in tax, accounting, legal, financial, or other domains where accuracy and explainability are critical.
Benefits
- Medical, dental, and vision plans.
- 401(k), FSA/HSA, commuter benefits, and tuition assistance.
- Vacation, sick time, and paid parental leave.
- Applicants may be required to appear onsite at a Wolters Kluwer office for part of the interview process.
Tech Stack
Categories
About Wolters Kluwer
Wolters Kluwer builds subscription software, data, and workflow solutions for professionals in healthcare, tax and accounting, legal and regulatory, finance and compliance, and ESG. Its portfolio spans SaaS applications and expert content that support compliance, decision-making, and operational processes. Founded in 1836 and headquartered in Alphen aan den Rijn, the Netherlands, it is publicly traded on Euronext Amsterdam (WKL) and reported 2025 annual revenue of €6.1 billion.