11 hours ago
Kitchener, CanadaSenior
Responsibilities
- Own end-to-end quality strategy, automation development, execution, and release qualification for agentic features and workflows.
- Build automation tooling and evaluation frameworks for AI/LLM-driven systems, including prompt flows, agent orchestration, tool integrations, benchmarking, and validation.
- Create automated coverage for critical AI workflows and integrate quality checks into CI/CD pipelines with fast PR feedback.
- Build LLM observability and debugging tools for prompt tracing and response analysis.
- Partner with Applied AI teams on prompt engineering, model selection, and evaluation strategies.
- Design performance and load tests for AI services, including latency, throughput, and cost efficiency.
- Identify and mitigate hallucination, bias, safety, and edge-case risks.
- Define and track AI quality KPIs such as task success rates, precision, recall, and latency.
- Participate in architecture and design reviews and mentor engineers on AI quality engineering practices.
Requirements
- 6+ years of experience in software engineering or SDET roles with an emphasis on software development.
- Strong programming skills in Python, Java, or JavaScript.
- Experience testing distributed, cloud-native SaaS systems and APIs and building scalable automation frameworks.
- Hands-on exposure to LLMs or AI/ML systems, including platforms such as OpenAI, Claude, or Gemini.
- Experience with non-deterministic systems, probabilistic testing, AI evaluation techniques, and human-in-the-loop validation.
- Experience with CI/CD pipelines such as Jenkins or GitHub Actions.
- Experience coding with AI agents to accelerate development and improve code quality.
- Bachelor’s degree in Computer Science or equivalent practical experience.
- Strong collaboration skills across distributed teams and time zones.
Benefits
- Competitive base salary of $151,000–$175,000 CAD for exceptional talent based in Ontario.
- Competitive benefits, perks, training, and opportunities for growth.
- Access to company AI tools and a collaborative, inclusive office environment.
- The position is based at one of Dialpad’s Canadian offices.
Tech Stack
ElasticsearchGitHub ActionsGoogle Cloud PlatformJavaJavaScriptJenkinsKubernetesPythonReactRedisVue.js
Categories
About Dialpad
Dialpad builds a cloud communications platform for businesses that unifies VoIP calling, video meetings, messaging, and AI-enabled contact center and sales tools. It sells subscription SaaS for business phone systems, collaboration, and customer support, including live transcription, analytics, and agent assist. Founded in 2011 and headquartered in San Ramon, California, the privately held company serves enterprises across industries, with customers such as Netflix and Randstad.