2 days ago
Remote, BoliviaSenior
Responsibilities
- Build automated evaluation frameworks for AI agents, RAG solutions, prompts, tools, workflows, and model outputs.
- Create benchmark datasets, golden responses, regression suites, evaluation metrics, and measurable release thresholds.
- Test AI behaviors including hallucinations, grounding, citation accuracy, prompt injection, tool selection, and multi-agent coordination.
- Develop agent-powered test-case generation, API and workflow validation, intelligent UAT, and UI/API/integration/performance/end-to-end automation.
- Implement CI/CD quality gates and monitor production AI quality, drift, reliability, and recurring failure patterns.
- Translate business requirements, workflows, and AI behaviors into measurable quality criteria.
Requirements
- At least 5 years of experience in Software Quality Engineering, Test Automation, or Reliability Engineering.
- Strong Python skills and experience building reusable automated testing frameworks.
- Hands-on experience with Playwright and/or Selenium.
- Strong experience with API, UI, integration, regression, end-to-end, and CI/CD-based test automation.
- Experience with Azure AI Foundry Evaluations, RAG evaluation, and prompt testing.
- Experience defining evaluation metrics, benchmark datasets, regression suites, and measurable quality thresholds.
- Knowledge of hallucination testing, grounding, citation accuracy, prompt injection, and agent behavior validation.
- Experience implementing CI/CD quality gates and production monitoring.
- Bachelor’s degree in Computer Science, Software Engineering, Information Systems, or a related field.
- Preferred experience testing Generative AI, RAG solutions, copilots, or AI agents.
- Preferred experience with LLM evaluation frameworks, advanced prompt testing, adversarial testing, and Responsible AI testing.
- Preferred experience developing agent-powered test generation, workflow validation, intelligent UAT assistants, and monitoring AI quality in production.
Benefits
- Remote work arrangement.
- Role available in Costa Rica, Peru, Colombia, and Bolivia.
About Encora
Encora provides software and digital product engineering services for technology companies and enterprises, including cloud-native development, data engineering, and QA. It operates a global delivery model and supports outsourced product development and managed engineering teams. Headquartered in Santa Clara, California, Encora is privately held and was acquired by Coforge; it previously raised $200 million in private equity funding in 2019.