Senior QA Analyst, AI & Machine Learning Systems
Keeper Security3 months ago
Remote, United StatesSenior / Mid Level
Responsibilities
- Design and execute test plans for AI and Machine Learning features.
- Develop Python-based automation and evaluation scripts.
- Test LLM-based features for accuracy and guardrail effectiveness.
- Validate data pipelines and model integration points.
- Conduct regression testing on ML models and AI workflows.
- Test enterprise-scale behavior analysis features.
- Validate security and privacy requirements for AI interactions.
- Collaborate with ML engineers and cross-functional teams.
- Analyze logs and metrics to detect anomalies and performance issues.
Requirements
- 3+ years of experience in QA or test automation.
- Hands-on experience writing Python automation scripts.
- Understanding of machine learning concepts.
- Experience testing LLM-integrated features.
- Familiarity with ML-related tooling like Pytest and Jupyter Notebooks.
- Experience with API testing and JSON validation.
- Ability to test nondeterministic systems.
- Strong analytical and troubleshooting skills.
- Experience testing cloud-integrated systems, preferably on AWS.
- Bachelor’s degree in Computer Science or equivalent experience.
Benefits
- Medical, Dental & Vision coverage.
- Employer Paid Life Insurance.
- Voluntary Short/Long Term Disability Insurance.
- 401K options (Roth/Traditional).
- Generous PTO plan including paid Bereavement/Jury Duty.
- Above market annual bonuses.