Software Engineer - Evals for Agents
Mercado Libre17 hours ago
Buenos Aires, ArgentinaMid Level
Responsibilities
- Evolve and maintain the online and offline agent evaluation platform.
- Develop production agent health capabilities including quality metrics, regression detection, observability, and alerts.
- Build offline experimentation tools for replaying real conversations, simulating users, and comparing model, prompt, and configuration versions.
- Create a continuous improvement cycle that turns evaluation results into actionable improvements for agent-owning teams.
- Develop APIs, a CLI, and integrations with AI-assisted development tools including Skills and MCPs.
- Collaborate with product, business, and technology teams to translate needs into impactful solutions.
- Stay current with generative AI models, agent evaluation, and related best practices.
Requirements
- Knowledge of microservice architecture, distributed systems, and asynchronous processing, including message queues, retry patterns, and idempotency.
- Solid knowledge of Go and/or Python, with the ability to learn additional languages.
- Knowledge of API design, RESTful APIs, and software development best practices.
- Knowledge of relational and non-relational databases.
- Familiarity with LLMs and their APIs, including GPT, Gemini, Claude, or others.
- Experience with evaluation frameworks, LLM-as-judge, LLM observability, AI platforms, agent development, data analysis, applied statistics, SQL, BigQuery, or experiment design is a plus.
Benefits
- Hybrid work arrangement.
- Work site in Buenos Aires, Argentina.
Categories
About Mercado Libre
Mercado Libre builds an e-commerce marketplace and fintech infrastructure for consumers and merchants across Latin America. It monetizes through marketplace transaction fees, payment processing (Mercado Pago), logistics and fulfillment (Mercado Envíos), advertising, and credit products. Founded in 1999 and headquartered in Buenos Aires, it operates in 18 countries and is publicly traded on Nasdaq under the ticker MELI.