Software Development Test Engineer (SDET), AI & Automation
Keeper Security3 hours ago
Remote, United StatesSenior
Responsibilities
- Partner with AI engineers, software engineers, and product managers to define test strategies, acceptance criteria, and automation requirements.
- Design and maintain automated testing frameworks for AI-powered applications, backend services, APIs, and supporting software systems.
- Develop deterministic and probabilistic validation strategies, including evaluation criteria, tolerances, scoring approaches, and quality thresholds for AI and LLM functionality.
- Build automated regression, integration, API, and end-to-end coverage alongside feature development.
- Create test harnesses and evaluation workflows for prompts, models, agents, tool-calling workflows, and AI-integrated applications.
- Integrate automated validation and quality gates into CI/CD pipelines.
- Maintain test datasets, mocks, fixtures, and synthetic data for repeatable validation.
- Analyze model and application failures, logs, telemetry, and system behavior to identify root causes and produce reproducible defect reports.
- Improve application testability, observability, diagnostic capabilities, test stability, and suite maintainability.
- Track test cases, execution results, and coverage using test management platforms.
- Contribute to AI testing methodologies, quality engineering standards, and automation practices across the AI organization.
- Use AI-assisted tools to improve automation development, test generation, debugging, analysis, and documentation.
Requirements
- At least 5 years of experience in SDET, software development, test automation, or another code-intensive quality engineering role.
- Hands-on experience testing AI/ML, LLM-powered, or model-integrated software systems.
- Strong understanding of deterministic software testing versus non-deterministic AI/ML validation.
- Strong programming skills in Python, Java, JavaScript/TypeScript, Rust, or another general-purpose programming language.
- Significant experience designing, developing, and maintaining automated test frameworks.
- Experience with automated functional, integration, API, and end-to-end test coverage.
- Experience testing REST APIs and distributed workflows with tools such as Postman or REST Assured.
- Experience integrating automated tests into CI/CD pipelines using GitHub Actions, Jenkins, GitLab CI, or similar technologies.
- Experience with TestRail or a similar test management platform.
- Experience with Git-based development workflows, code review, debugging, troubleshooting, logs, APIs, and application telemetry.
- Bachelor’s degree in Computer Science, Engineering, or a related technical field, or equivalent practical experience.
- U.S. Person status, defined as a U.S. citizen or lawful permanent resident, is required because of FedRAMP compliance requirements.
- Preferred experience includes LLMs, agentic systems, retrieval-augmented generation, tool calling, prompt-driven workflows, model evaluation, benchmarking, AI evaluation metrics, Rust, performance or adversarial testing, AI-assisted test generation, Docker, Kubernetes, cloud testing environments, and cybersecurity or identity software.
Benefits
- 100% remote work, with an optional hybrid schedule for candidates in the El Dorado Hills, California, or Chicago, Illinois metro areas.
- Medical, dental, and vision coverage, including domestic partnerships.
- Employer-paid life insurance and supplemental life insurance for employees, spouses, and children.
- Voluntary short- and long-term disability insurance.
- Traditional or Roth 401(k).
- Generous paid time off, including paid bereavement and jury duty leave.
Tech Stack
Categories
About Keeper Security
Keeper Security builds a cloud-based, zero-knowledge platform for password management, secrets management, privileged access management and secure remote connections for individuals and organizations. The privately held company, founded in 2011 and headquartered in Chicago, sells its KeeperPAM suite as subscription software, is published in 23 languages and sold in over 150 countries, and supports integrations with common identity providers and enterprise tech stacks.