Senior Software Engineer - Evals for Agents
Mercado Libre17 hours ago
Buenos Aires, ArgentinaSenior
Responsibilities
- Evolve and maintain the online and offline agent evaluation platform used by Mercado Libre teams.
- Develop production agent-health capabilities, including quality metrics, regression detection, observability, and alerting.
- Build offline experimentation tools for replaying real conversations, simulating users, and comparing model, prompt, and configuration versions.
- Create continuous-improvement workflows that turn evaluation results into actionable changes involving model selection, prompts, tools, and conversational flows.
- Build APIs, CLI tooling, and integrations with AI-assisted development tools such as Skills and MCPs.
- Collaborate with product, business, and technology teams to deliver high-impact solutions.
- Stay current with generative-AI, agent-evaluation, model, and observability practices.
Requirements
- Experience with distributed systems, microservice architecture, and asynchronous processing, including message queues, retry patterns, and idempotency.
- Solid knowledge of Go and/or Python, with the ability to learn additional languages.
- Knowledge of API design and software development practices such as SOLID, clean architecture, and design patterns.
- Knowledge of relational and non-relational databases.
- Familiarity with LLMs and APIs such as GPT, Gemini, and Claude.
- Preferred experience with evaluation frameworks, LLM-as-judge approaches, LLM observability, AI platforms, or agent development.
- Preferred knowledge of data analysis and applied statistics, including SQL, BigQuery, and experiment design.
Benefits
- Hybrid work arrangement.
- Work site is Buenos Aires, Argentina.
Categories
About Mercado Libre
Mercado Libre builds an e-commerce marketplace and fintech infrastructure for consumers and merchants across Latin America. It monetizes through marketplace transaction fees, payment processing (Mercado Pago), logistics and fulfillment (Mercado Envíos), advertising, and credit products. Founded in 1999 and headquartered in Buenos Aires, it operates in 18 countries and is publicly traded on Nasdaq under the ticker MELI.