Senior AI Engineer (Evaluations - Canvas Agent)
Instructure, Inc.about 4 hours ago
Responsibilities
- Design and maintain scalable LLM-as-a-judge evaluation pipelines.
- Create comprehensive grading rubrics and curate high-quality datasets.
- Optimize prompts for automated scoring to ensure quality evaluations.
- Participate in full-stack product development as needed.
- Integrate automated evaluations into CI/CD pipelines for efficient updates.
- Monitor evaluation metrics and translate quality signals into engineering tasks.
Requirements
- Hands-on experience designing and writing automated tests for LLMs.
- Strong background in Python, with a willingness to learn quickly.
- Professional experience in full-stack or backend engineering.
- Deep understanding of prompt engineering for classification and grading tasks.
- Familiarity with modern LLM observability and evaluation frameworks.
- Strong conceptual understanding of AI agents and multi-step workflows.
- Ability to translate subjective concepts into rigorous metrics.
- Collaboration skills to work across product and engineering teams.
Benefits
- Competitive compensation with participation in an ownership program.
- Flexible work culture with remote, hybrid, and in-office options.
- Generous time off, including local holidays and an annual recharge period.
- Comprehensive wellness programs and mental health support.
- Learning and development resources, including tuition reimbursement.
- Access to technology and tools for optimal work performance.
- Employee recognition program to celebrate contributions.
- Inclusive culture that values support and meaningful connections.