1 month ago
Canada (city not specified)Senior
Responsibilities
- Design evaluation frameworks covering the full agent execution lifecycle from intent and planning through retrieval, tool use, action, and business outcome.
- Evaluate agentic architectures, including single-agent, multi-agent, supervisor/router, planner-executor, sequential, parallel, tool-using, human-in-the-loop, and long-running workflows.
- Develop rigorous evaluations for retrieval precision, groundedness, citation accuracy, source quality, chunking, indexing, search, reranking, and permission-aware retrieval.
- Build automated evaluation infrastructure with golden and synthetic datasets, adversarial test suites, deterministic graders, LLM-as-a-Judge evaluation, human evaluation workflows, trace-based evaluation, and regression testing.
- Integrate evaluations into AI development and release pipelines so changes to models, prompts, retrieval, tools, and architectures can be measured before production.
- Analyze agent execution traces and create failure taxonomies covering model, retrieval, planning, tool, routing, memory, integration, policy, and orchestration failures.
- Establish production AI quality standards, release criteria, and metrics for task success, routing, tool selection, recovery, business outcomes, cost, and latency.
Requirements
- Strong experience with Python and software engineering.
- Experience with LLM application development, AI/ML evaluation and experimentation, automated testing, and data analysis.
- Experience with RAG, embeddings, hybrid search, reranking, tool or function calling, structured outputs, agent orchestration, and multi-agent workflows.
- Experience with AI observability and tracing, APIs, and enterprise integrations.
- Comfort with offline and online evaluations, deterministic graders, model-based graders, human evaluation, synthetic datasets, and adversarial testing.
- Experience with platforms or frameworks such as Agents SDK, LangGraph/LangChain, Microsoft AI Foundry, Amazon Bedrock, or similar agent platforms is valuable.
- Ability to design measurable experiments, analyze agent behavior and failures, and determine whether added agent complexity improves outcomes.
Benefits
- Competitive pay and benefits, with final compensation varying based on experience, skills, budget, location, bonuses, and incentives.
- Time away from work programs and comprehensive wellness initiatives.
- Recognition programs and opportunities for personal and professional growth.
- Volunteer days and community impact opportunities through Dayforce Cares.
- Equal opportunity workplace committed to diversity, employee health, fulfillment, and career development.
Tech Stack
Categories
About Ceridian
Ceridian, now operating as Dayforce, builds a cloud human capital management platform that unifies HR, payroll, talent, and workforce management for employers. It sells its software as a subscription and serves customers globally. Headquartered in Minneapolis, Minnesota (with a major presence in Toronto), the company traces the Dayforce brand to 2009 and is publicly traded on the NYSE after rebranding in 2024.