4 hours ago
Responsibilities
- Own end-to-end quality for agentic features and workflows, including strategy, automation development, execution, and release qualification.
- Build automation tooling and evaluation frameworks for AI/LLM-driven systems, prompt flows, agent orchestration, and tool integrations.
- Develop evals measuring response quality, accuracy, hallucination rates, task success, precision/recall, latency, and other AI quality KPIs.
- Drive at least 80% automation coverage for critical AI workflows using deterministic and probabilistic validation.
- Integrate AI quality checks into CI/CD pipelines with PR validation feedback cycles under 15 minutes.
- Build LLM observability and debugging tools, including prompt tracing and response analysis.
- Design and execute AI service performance and load tests covering latency, throughput, and cost efficiency.
- Identify and mitigate hallucination, bias, safety, and edge-case risks.
- Partner with Applied AI teams on prompt engineering, model selection, and evaluation strategies.
- Participate in design and architecture reviews to improve testability, observability, and resilience.
- Mentor engineers and advance AI quality engineering practices.
Requirements
- 6+ years of experience in software engineering or SDET roles with an emphasis on software development.
- Strong programming skills in Python, Java, or JavaScript, with Python preferred.
- Experience testing distributed, cloud-native SaaS systems and APIs.
- Experience coding with AI agents to accelerate development and improve code quality.
- Hands-on exposure to LLMs or AI/ML systems such as OpenAI, Claude, or Gemini.
- Understanding of non-deterministic systems and probabilistic testing approaches.
- Experience building scalable test frameworks and automation systems.
- Familiarity with AI evaluation techniques including benchmarking, golden datasets, and human-in-the-loop validation.
- Experience with CI/CD pipelines such as Jenkins or GitHub Actions.
- Bachelor’s degree in Computer Science or equivalent practical experience.
- Strong collaboration skills for working across distributed teams and time zones.
Benefits
- The position is based at one of Dialpad’s Canadian offices and reports to a QA Engineering Manager in the United States.
- The stated British Columbia base salary range is $150,500–$175,250 CAD, excluding bonus, equity, and benefits.
- Dialpad offers competitive benefits, training programs, AI tools, and opportunities for growth.
- Dialpad provides inclusive offices designed to support collaboration and connection.
- The company offers an equal-opportunity and anti-discrimination workplace.
Tech Stack
Categories
About Dialpad
Dialpad is the leading AI-powered communications intelligence platform creating human-first, AI-enhanced solutions that will drive the next wave of how businesses communicate with and serve their customers. Enterprise customers such as Randstad, RE/MAX, Nasdaq, Express Scripts, T-Mobile, Johns Hopkins, Motorola Solutions, Tractor Supply, and Netflix use Dialpad and its AI capabilities to deliver amazing customer experiences. Supported by notable investors such as Andreessen Horowitz, GV, ICONIQ Capital, and OMERS, Dialpad is a dynamic force in AI technology with a rapidly expanding presence. Visit dialpad.com to learn more.