Senior Software Engineer in Test (AI Agentic Systems)
Collective Health25 days ago
Base Salary
$99k - $124k/yr
Responsibilities
- Build and maintain a versioned library of 'Grounding Data' results.
- Design automated workflows to score factual grounding and policy compliance.
- Develop testing libraries to validate semantic equivalence and numerical accuracy.
- Programmatically verify that mandatory tools were invoked with correct arguments.
- Assert that agents respect defined priorities across architectural layers.
- Ensure every autonomous decision is traceable to specific SOP sentences.
- Trigger automated regression runs for every prompt or model update.
- Own the automated pairwise comparison process to detect logic drift.
- Build a mocking layer to simulate model responses for logic tests.
Requirements
- Expert in Python and pytest, with experience in building custom mocking frameworks.
- Hands-on experience with Vertex AI Experiments and Cloud Logging for trace analysis.
- Expert-level SQL and Pandas skills for identifying adjudication discrepancies.
- Ability to analyze 'System Instructions' and refine prompts based on test cases.
- Experience testing multi-layer systems involving RAG and function calling.
Benefits
- Mission-driven culture that values innovation and collaboration.
- Opportunities for professional development and mentorship.
- Flexible work arrangements and supportive work-life balance.
- Eligible for stock options and comprehensive health benefits.