QA Automation Engineer
BigBear.ai2 years ago
USA (city not specified)Mid Level
Responsibilities
- Translate product requirements, customer workflows, and feature interactions into testable outcomes and acceptance criteria.
- Design, implement, and maintain Playwright automation for critical user journeys, feature functionality, and regressions.
- Develop risk-based testing strategies based on customer impact, dependencies, and product priorities.
- Test conversational AI workflows including streaming responses, conversation history, file uploads, document retrieval, citations, model selection, and tool execution.
- Build AI evaluation datasets and scoring criteria for accuracy, grounding, instruction following, and unsafe-request handling.
- Integrate tests into CI/CD, investigate flaky tests, manage isolated test data, and provide failure diagnostics.
- Report defects with reproducible evidence, customer impact, severity, coverage gaps, and residual release risks.
- Document expected behavior, approved changes, testing decisions, and release-readiness evidence.
- Automate validation of backend APIs, authentication, billing and token behavior, model routing, AI workflows, file parsing, MCP and tool execution, agentic harnesses, and provider-facing API compatibility.
- Design cost-efficient AI workflow coverage while maintaining meaningful test signal.
- Automate security-sensitive validation for user isolation, permissions, input validation, sanitization, rate limits, replay prevention, safe errors, and layered controls.
- Use AI-assisted testing tools responsibly and maintain scalable test infrastructure, fixtures, and test data management.
Requirements
- 4+ years of QA automation engineering experience.
- Demonstrated experience building and maintaining Playwright tests, including fixtures, resilient locators, assertions, network handling, and trace-based debugging.
- Strong coding ability in TypeScript or JavaScript and experience reviewing changes through Git.
- Experience with API testing, CI/CD integration, test isolation, and diagnosing failures across browser, application, and backend boundaries.
- Ability to reason about complex requirements, explore edge cases, and balance testing depth with delivery priorities.
- Clear written and verbal communication with technical and nontechnical stakeholders.
- Experience testing authentication, authorization, permissions, and separation of customer data.
- Familiarity with validating Defense in Depth behavior without serving as a dedicated DevSec role.
- Strong troubleshooting and analytical skills and the ability to work independently and collaboratively.
- Ability to obtain a Department of Defense Secret clearance.
- Preferred experience testing LLM applications, retrieval-augmented generation, AI agents, model gateways, MCP tools, tool-using systems, or workflow automation platforms.
- Preferred Python experience for API testing, test utilities, test data generation, or AI evaluation workflows.
- Preferred experience with enterprise or government platforms, auditable test evidence, CI/CD pipelines, Docker, Kubernetes, observability, PostgreSQL, and SQL.
- Preferred familiarity with Google Vertex AI, AWS Bedrock, Microsoft Azure OpenAI, OpenAI GPT, Anthropic Claude, and Google Gemini.
- Preferred knowledge of FedRAMP, NIST AI RMF, and CMMC 2.0; an active DoD Secret clearance or higher is also preferred.
Tech Stack
Categories
About BigBear.ai
BigBear.ai develops AI-powered decision intelligence software, computer vision, and modeling/simulation tools for U.S. defense, intelligence, federal civilian, and critical-infrastructure markets. It sells software platforms and contract services, often deployed in secure government clouds, to support mission planning, logistics, and cybersecurity. Founded in 1988 and headquartered in McLean, Virginia, BigBear.ai is a public company listed on the NYSE under the ticker BBAI.