about 3 hours ago
Remote, Canada
Senior
H1B Sponsor
Responsibilities
- Design and ship an end-to-end AI evaluation framework.
- Define and instrument key performance metrics for AI agents.
- Build and maintain evaluation datasets and automated scoring pipelines.
- Architect and implement reusable agent infrastructure.
- Build and scale retrieval infrastructure and RAG pipelines.
- Make informed build vs. buy decisions for AI tools and frameworks.
- Own projects end-to-end and drive them to completion.
- Collaborate with engineering leads on evaluation strategy and methodology.
Requirements
- 5+ years of professional software engineering experience in production AI/ML systems.
- Deep hands-on experience with LLM-based systems and evaluation metrics.
- Proven ability to work with data and understand statistics.
- Experience building and operating agentic AI systems in production.
- Strong command of AI evaluation frameworks.
- Production-grade Python engineering skills.
Benefits
- Medical and dental insurance.
- Life, AD&D, and disability insurance.
- Paid parental leave and paid time off.
- Working remotely stipend and WFH office set-up stipend.
- Retirement plan and financial planning support.
- Learning and development budget.
Tech Stack
AWSDatadogGoogle Cloud PlatformMLflowPythonTypeScript
Categories
AI & MLData Science