Senior Software Development Engineer in Test
ComPsych Corporation22 days ago
USA (city not specified)Senior
Base Salary
$135k - $165k/yr
Responsibilities
- Define verification methods, evaluation thresholds, and release criteria for AI-powered features with variable outputs.
- Own the team's risk-based quality strategy, including automation, exploratory testing, and documented testing gaps.
- Build and maintain evaluation harnesses, fixtures, and shared automation patterns for model, prompt, and retrieval changes.
- Manage test execution speed, gating, failure classification, flake discipline, and actionable developer feedback.
- Define synthetic-data strategies, isolation patterns, and protections that prevent PHI from entering non-production environments.
- Set environment, observability, data, safety, security, performance, accessibility, and resilience verification requirements with Platform.
- Test AI failure modes including hallucination, unsafe output, and sensitive-data leakage.
- Establish reviewable agent conventions and boundaries for AI-assisted test generation and development.
Requirements
- Strong proficiency in a modern programming language and ability to reason about concurrency, asynchronous workflows, and failure propagation across services.
- Ability to trace failures across services using logs, traces, and deployment context and distinguish product, automation, data, and environment causes.
- Experience communicating residual risk and making release decisions without becoming a bottleneck.
- Demonstrated depth in at least one non-functional area such as performance, resilience, security, or accessibility.
- Required experience verifying LLM-based or other non-deterministic products where identical inputs can produce different valid outputs.
- Experience designing curated evaluation sets, rubrics, statistical pass thresholds, and human- or model-assisted grading.
- Experience detecting regressions across model, prompt, or retrieval changes and testing hallucination, unsafe output, and sensitive-data leakage.
- Preferred experience building shared testing frameworks and standards adopted by multiple teams.
- Preferred experience with synthetic-data strategy, deterministic isolation for sensitive data, failure injection, recovery testing, healthcare, regulated domains, or sensitive customer data.
Benefits
- Full benefits package including paid time off, medical, dental, vision, 401(k) with match, employee assistance program, and wellness program.
- The stated base salary range is $135,000-$165,000 USD.