5 days ago
Remote, Peru +4 moreSenior
Responsibilities
- Design and maintain automated unit, integration, API, and end-to-end test suites.
- Integrate automated testing into CI/CD pipelines and establish automated suites as release quality gates.
- Build AI evaluation frameworks using scored rubrics, LLM-as-a-judge approaches, and normalized pass thresholds.
- Maintain held-out evaluation datasets, measure AI behavior against ground truth, and report defect escape and evaluation results.
Requirements
- 4+ years of quality engineering experience focused on test automation.
- Hands-on experience with automated testing frameworks, preferably in the .NET ecosystem, plus strong API testing and automated test development experience.
- Experience integrating automated test suites into CI/CD pipelines.
- Understanding of testing strategies for non-deterministic and probabilistic AI systems, including threshold-based evaluation instead of exact-match assertions.
- Experience with test data, ground-truth datasets, and automated validation.
- Strong analytical, problem-solving, and English communication skills.
- Bachelor's degree in Computer Science, Software Engineering, Information Technology, or a related field, or equivalent professional experience.
- Preferred experience with LLM evaluation, rubric design, LLM-as-a-judge, golden or held-out datasets, Azure AI Foundry Evaluators, promptfoo, Ragas, or DeepEval.
- Preferred experience testing information extraction or classification pipelines using accuracy-based measurements, regulated-industry QA environments, and AI/ML testing methodologies.
Benefits
- Remote work mode.
- Role available in Costa Rica, Peru, Colombia, or Bolivia.
Tech Stack
.NET
Categories
About Encora
Encora provides software and digital product engineering services for technology companies and enterprises, including cloud-native development, data engineering, and QA. It operates a global delivery model and supports outsourced product development and managed engineering teams. Headquartered in Santa Clara, California, Encora is privately held and was acquired by Coforge; it previously raised $200 million in private equity funding in 2019.