Senior AI Systems Quality Engineer
Abacus Insights1 month ago
Responsibilities
- Build and ship production-grade automated validation frameworks, test harnesses, and evaluation pipelines across the AI lifecycle.
- Design and evolve an AI testing platform integrated with Databricks and MLflow for repeatable testing, traceability, and auditability.
- Create large-scale scenario-based test suites covering agentic workflows, edge cases, long-tail scenarios, and failure modes.
- Validate orchestration behavior, including tool use, memory, decision logic, and non-deterministic system behavior.
- Define system contracts, guardrails, safe-degradation patterns, and measurable quality signals for LLM systems.
- Integrate automated quality gates into CI/CD pipelines and ensure validation runs on model, prompt, and code changes.
- Build reusable libraries and components for consistent AI quality practices.
- Own aspects of AI release readiness by defining go/no-go criteria and measurable quality thresholds.
- Partner with AI, platform, security, delivery, and domain teams to establish production-relevant quality criteria and validation scenarios.
Requirements
- 7+ years of software engineering experience, primarily in backend or platform systems.
- Production experience designing and implementing AI testing automation rather than only executing tests.
- Experience building custom validation, evaluation, or testing frameworks for complex distributed systems.
- Strong proficiency in Python and/or TypeScript within modern AI engineering stacks.
- Hands-on experience with LLM-based or agentic systems and non-deterministic behavior.
- Experience with AI testing at scale, regression frameworks, long-tail evaluation, and broad test coverage.
- Deep understanding of integrating automated tests and quality gates into CI/CD deployment pipelines.
- Solid understanding of AWS cloud-native architectures.
- Experience engineering for quality, reliability, governance, and safety in regulated or mission-critical environments.
- Experience with non-deterministic output evaluation, drift detection, bias and fairness testing, and robust regression strategies.
- Ability to define and operationalize trust thresholds involving query accuracy, hallucination limits, explainability, and PHI-safe behavior.
- Experience working with domain experts to create business-relevant production validation scenarios.
- Preferred experience with Databricks-native environments, Medallion architecture, MLflow, observability tools, LLM evaluation techniques, guardrails, policy enforcement, and AI performance or cost regression testing.
- Preferred formal AI/ML systems training or certification, such as ISTQB AI Testing, AWS ML Specialty, or Google ML Engineer.
- Preferred experience versioning and testing prompts, agent behaviors, and orchestration logic, and using AI systems to generate diverse or adversarial test datasets.
Benefits
- Unlimited paid time off.
- Work from anywhere.
- Comprehensive health coverage with multiple plan options.
- Equity for every employee.
- Growth-focused development environment.
- One-time home office setup allowance.
- Monthly cell phone allowance.
Tech Stack
Categories
About Abacus Insights
Abacus Insights builds a cloud-based, AI-ready data platform for health plans that ingests, standardizes, and governs payer and clinical data to power analytics and workflows. Sold as SaaS with pre-built connectors, it supports CMS interoperability, risk adjustment, HEDIS/Stars quality improvement, and payment integrity. Founded in 2017 and headquartered in Boston, the privately held company reports 35 payer customers covering 65 million members.