15 hours ago
Responsibilities
- Build and maintain production-grade test infrastructure, AI/ML evaluation systems, agentic validation harnesses, release orchestration tooling, observability systems, and performance testing platforms.
- Design agentic test harnesses that interact with applications, browsers, APIs, databases, logs, telemetry, and CI/CD systems to execute and validate complex workflows.
- Create evaluation frameworks for non-deterministic systems using deterministic assertions, semantic and model-based evaluation, regression datasets, and human review.
- Implement structured logging, metrics, distributed tracing, and execution timelines to diagnose automated and agentic workflow outcomes.
- Develop systems that distinguish product, test, infrastructure, and AI/model reasoning failures and correlate them with operational evidence.
- Design test automation frameworks, migrate from Cypress to Playwright, and own test environment provisioning, parallelization, and CI/CD integration.
- Apply AI to test generation, flaky-test triage, failure clustering, and release-risk analysis with evidence-based guardrails.
- Partner with Software Engineering, Quality Engineering, SRE, and Platform teams and mentor engineers on test architecture and AI-assisted engineering.
Requirements
- At least 5 years of experience as an SDET or software engineer building test infrastructure, developer tooling, platform engineering, observability, or release engineering systems.
- Strong programming ability in JavaScript and TypeScript, including backend services, libraries, frameworks, and production infrastructure.
- Strong fundamentals in system design, distributed systems, APIs, asynchronous systems, and production operations.
- Experience scaling CI/CD systems, release orchestration, test infrastructure, developer platforms, or observability and telemetry systems.
- Hands-on experience with LLMs, AI-assisted engineering, or agentic systems, including tool/function calling, agent workflows, and multi-step AI workflows.
- Experience or familiarity with LLM and agent evaluation, evaluation datasets, deterministic evaluators, model-based evaluators, and probabilistic systems.
- Strong understanding of the testing pyramid and validation across unit, integration, API, UI, performance, and agentic layers.
- Ability to mentor engineers and raise technical quality standards across teams.
Benefits
- Remote-first work arrangement with the role based in Austin, Texas and remote candidates considered.
- Comprehensive health plans, a 401k program, commuter benefits, professional development, parental leave, and flexible time off.
- Robust online onboarding, biweekly all-hands meetings, and internal virtual events.
Tech Stack
Apache JMeterAWSCypressDatadogGitHub ActionsGoogle CloudHarnessJavaScriptJenkinsJestKubernetesMongoDBNode.jsPostmanSpinnakerTypeScript
Categories
About Qualia
Qualia builds cloud software that connects title and escrow companies, mortgage lenders, and real estate agents to manage home closings end to end. Its SaaS platform covers title production, escrow accounting, digital mortgage workflows, e-signatures, remote notary, and borrower/agent collaboration for SMB and enterprise customers. Founded in 2015 and headquartered in San Francisco, the privately held company also has offices in Austin, TX, and Superior, CO.