5 days ago
Toronto, CanadaMid Level
Responsibilities
- Curate and maintain golden datasets containing documents, user queries, and expected agent trajectories.
- Design and implement automated continuous evaluation pipelines for agent accuracy, latency, token spend, and fallback reliability.
- Trace and audit multi-step agent reasoning and execution paths to identify deviations and cascading errors.
- Optimize system prompts, context windows, few-shot examples, tool use, function calling, and external API or database interactions.
- Collaborate with AI solution leads and agent developers to incorporate evaluation findings into reusable agent components.
- Monitor production agents for semantic drift, prompt injection, infinite loops, hallucinations, and other reliability issues.
- Implement production guardrails and optimize domain knowledge bases and RAG pipelines.
Requirements
- 3+ years of professional experience in software quality engineering, test automation, or data/ML engineering.
- Dedicated experience with LLM testing, prompt tuning, or orchestration patterns during the last 1–2 years.
- Hands-on experience with LLM orchestration frameworks such as LangGraph, ADKs, or specialized internal SDKs.
- Deep understanding of JSON schema design for LLM tool calling, function calling, and structured outputs.
- Strong experience writing automated test scripts, particularly in Python, and debugging asynchronous, non-deterministic systems using tracing and observability concepts.
- Experience with AI evaluation and observability platforms is advantageous.
- Live production experience testing agentic workflows and generative AI solutions is advantageous.
- Familiarity with Google Cloud AI services, Vertex, Gemini Enterprise Agent Platform, UiPath, or Maestro is advantageous.
- Experience securely generating synthetic data with LLMs for edge-case testing is advantageous.
- Proficiency in Python or TypeScript and understanding of asynchronous programming, API design, and microservices architecture are advantageous.
Tech Stack
Categories
About Equifax
Equifax is a global data and analytics company and one of the three major U.S. credit bureaus, providing credit reports, risk scores, fraud prevention, and identity protection to lenders, employers, and government agencies. It sells data, analytics platforms, and consumer monitoring subscriptions, and offers employment verification through The Work Number. Founded in 1899 and headquartered in Atlanta, it is publicly traded on the NYSE and operates in 24 countries.