1 day ago
Bogotá, Colombia or Guatemala City, GuatemalaSenior
Responsibilities
- Own the end-to-end test strategy for an AI voice virtual agent across LLM, speech, and telephony layers.
- Design and execute complex conversational, call-flow, failure-recovery, and exploratory test scenarios.
- Build and maintain evaluation frameworks using golden datasets, semantic or rubric-based assertions, and regression detection.
- Adversarially test for hallucinations, incorrect grounding, guardrail bypasses, prompt injection, and unauthorized data disclosure.
- Prepare SIP devices and call scenarios, automate call origination, and diagnose issues across LLM, ASR, TTS, media, and routing layers.
- Extend automated coverage and integrate test suites into release pipelines as quality gates.
- Define quality metrics and communicate release risks using data and documented evidence.
- Maintain traceable test coverage in the TMS and participate fully in Scrum ceremonies and team artifacts.
Requirements
- 5+ years of QA experience with recent hands-on ownership of a complex integrated product.
- Practical SIP/VoIP testing experience covering call flows, SIP signaling, media handling, softphones, or SIP trunking, including troubleshooting with SIP traces or packet captures.
- Proficiency in Python sufficient to build and maintain automation frameworks independently.
- Experience testing non-deterministic or probabilistic systems and designing assertions for variable outputs.
- Strong analytical skills for defining quality metrics and supporting release decisions with data.
- Hands-on experience with formal TMS tools and Jira.
- Professional fluency in spoken and written English with reliable overlap with US Eastern business hours.
- Preferred experience includes LLM or conversational AI testing, ASR and TTS evaluation, RAG systems, WebRTC and media-layer testing, IVR or contact-center platforms, multilingual testing, and prompt engineering literacy.
Benefits
- Remote work opportunity for candidates in LATAM.
- B2B employment arrangement with gross compensation.
- Long-term growth perspective, annual performance review, and continuous learning opportunities.
- Company support for professional training, seminars, conferences, and English classes.
- Good hardware, referral program, and work with a talented team.
