2 days ago
Responsibilities
- Build and version golden question suites with expected sources, passages, and acceptance thresholds for relevance, grounding, and citation accuracy.
- Create negative entitlement tests that verify unauthorized identities cannot retrieve restricted passages.
- Develop freshness and retirement tests for indexing lag and removal of retired content.
- Automate evaluation runs after content, configuration, and model changes, publishing scores, trends, and regression alarms to a quality dashboard.
- Define QA gates for remediated content and run evaluation suites before pilot changes.
- Report misses and root causes across content, chunking, ranking, and permissions to the appropriate owners.
- Write the evaluation runbook and train client stewards and platform owners to maintain questions, run suites, and interpret regressions.
Requirements
- 3+ years of experience in software quality engineering or test automation with Python, including scheduled CI test harnesses.
- Hands-on production evaluation of search or RAG systems, including ground-truth creation, relevance and grounding scoring, and drift tracking.
- Experience designing security-focused negative tests for authorization, entitlements, and data exposure.
- Ability to work with AWS by invoking Bedrock and OpenSearch APIs, reading CloudWatch, and deploying scheduled jobs with infrastructure code.
- Strong written reporting skills for communicating evaluation results to client stewards.
- Preferred: AWS Certified AI Practitioner or Machine Learning Engineer Associate certification.
- Preferred: experience with RAG evaluation libraries and LLM-as-judge scoring with human calibration.
- Preferred: contact center or knowledge management domain experience.
- Preferred: familiarity with SOC 2 or ISO 27001 evidence practices.
Benefits
- Work on real-world AI-driven projects across multiple industries and client environments.
- Collaborate with a global team across continents and cultures.
- Inclusive environment focused on continuous learning, innovation, and ethical AI standards.
- Part-time allocation of 50% across weeks 3–10, reporting to the POD Lead and working with the retrieval engineer.
About Solvd
Solvd is a privately held software engineering and consulting firm that builds AI, data, cloud, application development, and quality engineering solutions for enterprises across technology, ecommerce, retail, fintech, hospitality, and banking. Headquartered in Walnut Creek, California, it is a portfolio company of Siguler Guff & Company. Following its acquisition of Tooploox, Solvd delivers end-to-end engagements from strategic advisory to custom AI development and enterprise-scale implementation for Fortune 500 clients.