Senior AI Engineer (Evaluations - Canvas Agent)
Instructure, Inc.2 months ago
Responsibilities
- Design and maintain scalable LLM-as-a-judge pipelines that score Canvas Agent actions, responses, and tool usage across complex educational workflows.
- Create grading rubrics and curate real and synthetically generated golden datasets for agent testing and baselining.
- Engineer and iterate on judge prompts to improve scoring quality and reliability.
- Contribute to full-stack product development, including AI features, UI components, and application architecture.
- Integrate automated evaluations into CI/CD pipelines to accelerate safe product iteration.
- Monitor evaluation metrics, identify failure modes, hallucination rates, and regressions, and turn findings into actionable engineering work.
Requirements
- Proven hands-on experience designing and implementing automated LLM-as-a-judge evaluations in real-world scenarios.
- Strong Python background or demonstrated ability to learn Python quickly.
- Professional full-stack or backend engineering experience; TypeScript and Node.js experience is a plus.
- Deep understanding of prompt engineering for classification, extraction, and grading, including mitigation of position and verbosity bias.
- Familiarity with LLM observability and evaluation frameworks such as LangSmith, Braintrust, Ragas, or promptfoo.
- Strong conceptual understanding of agentic architectures, planning, and tool calls, including ReAct and tool-use APIs.
- Ability to translate subjective qualities such as helpfulness and tone into rigorous, trackable metrics.
- Ability to collaborate across product and engineering teams and align evaluation criteria with user needs.
Benefits
- Competitive compensation and ownership program for full-time employees.
- Remote, hybrid, and in-office work arrangements vary by role, team, and location.
- Generous time off, local holidays, and an annual late-December “Dim the Lights” period based on departmental needs.
- Wellness programs and mental health support.
- Learning and development resources, professional development tools, and tuition reimbursement.
- Motivosity employee recognition program.
- Employees receive the technology and tools needed for their work.
- Background check and identity verification are required as part of the hiring process.
Tech Stack
Categories
About Instructure, Inc.
🎉 Show us how you’re experiencing #InstructureCon2026! We’d love to see the moments that made your event memorable. Share your photos, favorite takeaways, and highlights using #InstructureCon2026 and tag @Instructure. Copyright © 2016, Instructure, Inc. All rights reserved.