2 days ago
Responsibilities
- Design and execute AI-agent evaluation frameworks using test datasets, LLM-as-a-Judge methodologies, human reviews, and regression testing.
- Build, configure, and optimize AI guardrails and moderation systems for regulated financial-services and credit-product domains.
- Develop and continuously improve prompts based on evaluation results and production trace analysis.
- Create and maintain MCP servers and tool integrations connecting AI agents with internal data and services.
- Support fine-tuning and evaluation of language models using domain-specific datasets.
- Prototype and validate AI workflows using Jev and Laya models for classification, routing, and verification.
- Monitor AI quality, latency, performance, and costs through Langfuse and implement improvements.
- Collaborate with Engineering, Data Science, and Compliance teams to deliver reliable, secure, and compliant AI solutions.
Requirements
- 1–3+ years of experience in Machine Learning, Applied AI, Data Science, or Software Engineering.
- Experience with LLMs and AI-powered applications.
- Knowledge of LLM evaluation techniques, benchmark datasets, scoring frameworks, and LLM-as-a-Judge approaches.
- Understanding of multi-turn AI-agent behavior and tool-calling workflows.
- Experience with prompt engineering, structured outputs, and prompt optimization.
- Familiarity with moderation, prompt-injection prevention, jailbreak mitigation, and AI guardrails.
- Hands-on experience with AWS cloud services, preferably Amazon Bedrock.
- Experience with Langfuse or similar observability and evaluation platforms for LLM applications.
- Strong Python programming skills.
- Experience developing REST APIs using FastAPI or similar frameworks.
- Knowledge of unit testing, code reviews, and Git-based development workflows.
- Advanced written and verbal English communication skills.
- Preferred: MCP server development, LLM fine-tuning with LoRA or full fine-tuning, Small Language Models, model distillation, deployment strategies, Jev and Laya, regulated-industry experience, and AI evaluation and production-monitoring practices.
Benefits
- Remote work arrangement in LATAM.
Categories
About Encora
Encora provides software and digital product engineering services for technology companies and enterprises, including cloud-native development, data engineering, and QA. It operates a global delivery model and supports outsourced product development and managed engineering teams. Headquartered in Santa Clara, California, Encora is privately held and was acquired by Coforge; it previously raised $200 million in private equity funding in 2019.