Senior Software Engineer in Test (AI Agentic Systems)
Collective HealthBase Salary
$99k - $136k/yr
Responsibilities
- Build and maintain a versioned golden-set library of grounding data and ground-truth results with senior claims examiners.
- Design automated model-as-a-judge workflows and custom rubrics for factual grounding and policy compliance.
- Develop semantic assertion frameworks that validate semantic equivalence and numerical accuracy in agent outputs.
- Audit Vertex AI traces to verify that mandatory MCP tools are invoked with correct arguments.
- Validate orchestration priorities across data and knowledge, orchestration, agentic reasoning, and tooling layers.
- Ensure autonomous decisions are traceable to specific SOP sentences and live API data points.
- Integrate automated regression runs with Vertex AI Prompt Management for every prompt or model update.
- Own automated side-by-side comparisons between new and production agent versions to detect logic drift.
- Build Vertex AI and ADK mocking infrastructure for high-volume, low-cost logic testing.
Requirements
- 5+ years of experience in software testing and development.
- Expert Python and pytest skills, including custom mocking frameworks for external APIs such as Vertex AI and ADK.
- Hands-on experience with Vertex AI Experiments, Auto-SxS, and Cloud Logging for trace analysis.
- Expert-level SQL with BigQuery and Pandas skills for comparing large datasets and identifying adjudication discrepancies.
- Ability to analyze system instructions and refine prompts based on failed test cases.
- Experience testing multi-layer systems involving RAG with Vertex AI Search, state management with LangGraph, and function calling.
- Familiarity with healthcare claims adjudication concepts such as pend reason codes, COB, eligibility, and stop-loss is preferred.
- Understanding of HIPAA/PHI handling and preparation of test evidence for DOL and DOI regulators is preferred.
- Experience with shadow-mode monitoring and comparing agent decisions against human expert baselines is preferred.
Benefits
- Hybrid work based in Lehi, Utah or Plano, Texas, with at least two weekdays in the office.
- Health insurance, 401(k), and paid time off.
- Professional development through internal mobility, mentorship programs, and tailored courses.
- Flexible work arrangements and work-life balance support.
Categories
About Collective Health
Collective Health is the leading independent third party administrator (TPA) that brings together plan administration, cost management, and taking great care of your people—all in one place. Conventional wisdom in health benefits says you must choose: better or safe. Collective Health allows you to have both. For self-funded employers, Collective Health provides a unified foundation with Collective TPA™, My Collective®, Care Navigation, Collective Compass®, and Partner Collective®—combining technology, clinical expertise, and a flexible partner ecosystem to simplify administration, improve decision-making, and guide members through their healthcare journeys. Founded in 2013, Collective Health serves more than 800,000 members from organizations including Red Bull, The Wonderful Company, Boot Barn, Live Nation, Driscoll’s, Uber, and Zoom. The company is headquartered in San Francisco, California, with additional locations in Plano, Texas, and Lehi, Utah. *We value respectful and constructive conversation. Comments that are offensive, inappropriate, or unrelated to the discussion may be removed.*