2 days ago
Remote, Peru +3 moreSenior
Responsibilities
- Build automated evaluation frameworks for AI agents, RAG solutions, prompts, tools, workflows, and model outputs.
- Create benchmark datasets, golden responses, regression suites, evaluation metrics, and measurable release thresholds.
- Test hallucinations, grounding, citation accuracy, prompt injection, tool selection, and multi-agent coordination.
- Develop agent-powered test-case generation, API and workflow validation, intelligent UAT, and UI, API, integration, performance, and end-to-end automation.
- Implement CI/CD quality gates and monitor production AI quality, drift, reliability, and recurring failure patterns.
- Translate business requirements, workflows, and AI behaviors into measurable quality criteria.
Requirements
- 5+ years of experience in Software Quality Engineering, Test Automation, or Reliability Engineering.
- Strong Python skills and experience building reusable automated testing frameworks.
- Hands-on experience with Playwright and/or Selenium.
- Strong experience with API, UI, integration, regression, end-to-end, and CI/CD-based test automation.
- Experience with Azure AI Foundry Evaluations, RAG evaluation, prompt testing, evaluation metrics, benchmark datasets, regression suites, and quality thresholds.
- Knowledge of hallucination testing, grounding, citation accuracy, prompt injection, and agent behavior validation.
- Experience implementing CI/CD quality gates and production monitoring.
- Bachelor’s degree in Computer Science, Software Engineering, Information Systems, or a related field.
- Preferred qualifications include experience with Generative AI, RAG solutions, copilots, AI agents, LLM evaluation frameworks, advanced prompt testing, agent-powered test generation, adversarial testing, Responsible AI testing, and production AI quality monitoring.
Benefits
- Remote work arrangement.
- Role is based in Costa Rica, Peru, Colombia, or Bolivia.
Tech Stack
PythonSelenium
Categories
About Encora
Encora provides software and digital product engineering services for technology companies and enterprises, including cloud-native development, data engineering, and QA. It operates a global delivery model and supports outsourced product development and managed engineering teams. Headquartered in Santa Clara, California, Encora is privately held and was acquired by Coforge; it previously raised $200 million in private equity funding in 2019.