Testing Engineer
Robots and Pencils19 days ago
Colombia (city not specified)Mid Level
Responsibilities
- Write and maintain automated UI, API, integration, and ML workflow tests.
- Validate AI/ML model outputs, data pipelines, datasets, and AI workflows against defined quality criteria.
- Perform LLM evaluation, RAG retrieval and grounding checks, model drift monitoring, prompt regression testing, and AI safety and red-teaming.
- Execute automated test suites, analyze results, and document and track defects.
- Integrate automated tests into CI/CD pipelines and contribute to continuous quality practices.
- Collaborate with developers, ML engineers, and data scientists on quality strategy and delivery goals.
- Provide testing reports, communicate quality status, own testing responsibilities end-to-end, and improve test coverage and QA processes.
Requirements
- At least 2 years of experience in test automation or AI/ML testing, with comfort in one area and working knowledge of the other.
- Proficiency in Python, JavaScript, Java, or a similar programming language, with the ability to maintain test code independently.
- Experience with UI, API, or ML workflow test automation frameworks.
- Understanding of AI/ML concepts and the ML lifecycle, including the ability to validate model outputs.
- Experience with CI/CD pipelines, version control, API testing, data validation, bug tracking, and test management tools.
- Understanding of SDLC, STLC, and Agile methodologies.
- Familiarity with LLM evaluation, RAG quality and grounding, model drift, prompt regression testing, AI safety, and red-teaming.
- Demonstrable day-to-day use of AI-forward tools such as Claude and Cursor.
- Strong problem-solving skills, sound judgment, curiosity, and the ability to work through ambiguous testing challenges.
- Experience with mobile automation, performance testing, MLOps pipelines, or basic statistics is preferred.
Tech Stack
Categories
About Robots and Pencils
Robots & Pencils is an applied AI engineering firm that designs, builds, and runs production AI systems and agentic co-workers for enterprise operations, with a focus on AWS. The privately held company, founded in 2009 and headquartered in Chagrin Falls, Ohio, provides consulting and implementation services spanning data readiness, cloud modernization, and AI enablement. It is an AWS Advanced Tier Services Partner with delivery teams across the U.S., Canada, Eastern Europe, and Latin America.