2 months ago
Vancouver, CanadaSenior
Responsibilities
- Own end-to-end quality for agentic features and workflows, including strategy, automation development, execution, and release qualification.
- Build automation tooling and evaluation frameworks for AI/LLM-driven systems, prompt flows, agent orchestration, and tool integrations.
- Develop evals measuring response quality, accuracy, hallucination rates, task success, precision/recall, latency, and other AI quality KPIs.
- Drive at least 80% automation coverage for critical AI workflows using deterministic and probabilistic validation.
- Integrate AI quality checks into CI/CD pipelines with PR validation feedback cycles under 15 minutes.
- Build LLM observability and debugging tools, including prompt tracing and response analysis.
- Design and execute AI service performance and load tests covering latency, throughput, and cost efficiency.
- Identify and mitigate hallucination, bias, safety, and edge-case risks.
- Partner with Applied AI teams on prompt engineering, model selection, and evaluation strategies.
- Participate in design and architecture reviews to improve testability, observability, and resilience.
- Mentor engineers and advance AI quality engineering practices.
Requirements
- 6+ years of experience in software engineering or SDET roles with an emphasis on software development.
- Strong programming skills in Python, Java, or JavaScript, with Python preferred.
- Experience testing distributed, cloud-native SaaS systems and APIs.
- Experience coding with AI agents to accelerate development and improve code quality.
- Hands-on exposure to LLMs or AI/ML systems such as OpenAI, Claude, or Gemini.
- Understanding of non-deterministic systems and probabilistic testing approaches.
- Experience building scalable test frameworks and automation systems.
- Familiarity with AI evaluation techniques including benchmarking, golden datasets, and human-in-the-loop validation.
- Experience with CI/CD pipelines such as Jenkins or GitHub Actions.
- Bachelor’s degree in Computer Science or equivalent practical experience.
- Strong collaboration skills for working across distributed teams and time zones.
Benefits
- The position is based at one of Dialpad’s Canadian offices and reports to a QA Engineering Manager in the United States.
- The stated British Columbia base salary range is $150,500–$175,250 CAD, excluding bonus, equity, and benefits.
- Dialpad offers competitive benefits, training programs, AI tools, and opportunities for growth.
- Dialpad provides inclusive offices designed to support collaboration and connection.
- The company offers an equal-opportunity and anti-discrimination workplace.
Tech Stack
Categories
About Dialpad
Dialpad builds a cloud communications platform for businesses that unifies VoIP calling, video meetings, messaging, and AI-enabled contact center and sales tools. It sells subscription SaaS for business phone systems, collaboration, and customer support, including live transcription, analytics, and agent assist. Founded in 2011 and headquartered in San Ramon, California, the privately held company serves enterprises across industries, with customers such as Netflix and Randstad.